Speaker: Ulrike Genschel, Associate Professor, Department of Statistics, Iowa State University

Title: A Modified t-test for Treatment Means in Unreplicated Classroom Comparisons

Abstract: Discipline-based education research (DBER), with a focus on evidence-based teaching, has grown immensely over the last decades.  A common interest in DBER studies is identifying superior pedagogical approaches using rigorous and scientific methodology. Researchers may have few classrooms available when comparing classroom-level treatments or conditions so that one classroom per treatment is not uncommon in many DBER studies. 

Because data and analysis options are then limited, an approach often seen in the DBER literature is to compare treatment means with a two-sample t-test applied to student-level responses from each classroom.  This strategy, however, carries particular risks for statistical inference, where p-values can be misleading to an extent that is often under-appreciated and also much worse than possibly overstating practical significance. 

We demonstrate that, even in the absence of any treatment difference, a mathematical guarantee exists that the p-value from a standard two-sample t-test applied to student-level responses in this setting can be made arbitrarily close to zero with probability 1, simply as an artifact of sufficient student enrollment.

Existing options to remedy the t-test, as we review, are typically intractable. As a more reasonable assessment of evidence, we propose a modified two-sample t-test for comparing treatment means, which involves a smoothing step to account for classroom-level experimental error rather than ignoring this and possible correlations among student responses.  Our numerical studies show that the modified t-test performs better than the standard t-test in controlling false rejection rates. The method is also illustrated with applications to several real data sets from educational studies.