Speaker: ZhiMin Xiao (University of Exeter)

Abstract: Analyses of social interventions need to produce evidence that is relevant to different groups of people in a society. When such a group is not the target group of an intervention, this is called subgroup analysis, even when the group of interest is pre-specified prior to data collection. Amongst statisticians, subgroup analysis is often regarded as a statistical malpractice, as its findings are often underpowered, unreliable, and can be prone to overinterpretation at best, or misleading at worst. Meanwhile, researchers would be criticised for generating irrelevant evidence and accused of wasting research money if they do not conduct relevant subgroup analysis. As a result, “they are damned if they do, and damned if they don’t” (Petticrew et al., 2012). 

In this study, we estimated intervention effects for Free School Meal (FSM) pupils in English schools, which is a pre-specified subgroup in most educational interventions funded by the Education Endowment Foundation (EEF) in England. Specifically, we first ran a treatment-FSM interaction test in each and every outcome to see if the difference-in-effects is statistically significant between FSM and Non-FSM pupils. We then calculated separate effect sizes within the two subgroups. Finally, we examined the p-values from the interaction tests and compared the overall effect sizes for both FSM and Non-FSM pupils with the two separate subgroup estimates. We found that conventional interaction tests can produce self-contradictory results. To help solve the problem, we propose a new approach, Pupil Advantage Index (PAI), as an alternative to subgroup analysis and apply it to real RCT data extracted from the EEF data archive. We demonstrate that PAI does not just indicate where an intervention worked and by how much in existing trials, but it can also be utilised to optimise treatment recommendation for future interventions.