Doing more with less: enabling efficiency in education research with the teacher-education dataset (TED)

New teaching methods and leadership approaches are needed to improve outcomes for pupils, especially for pupils from disadvantaged backgrounds. Before deciding to try something new – to make an ‘intervention’ with teachers and leaders – school and trust leaders rightly want evidence that the planned changes have been effective elsewhere.

This presents a problem. Trialling and assessing the effectiveness of new approaches to teaching is often an expensive endeavour, regularly involving over 100 schools to answer one question with the ‘gold standard’ of a randomised controlled trial. This brings obvious challenges to researchers whose job it is to evaluate interventions, in terms of the time, effort and expense associated with research projects of this size.

It is possible that the National Institute of Teaching’s TED may soon make it possible to evaluate interventions with the same level of precision while working with far fewer schools. As part of work funded by the Nuffield Foundation, we explored this possibility to see if we might be able to reduce the cost of generating valuable, sector-wide insights into teaching practices through intelligent use of data.

What we did

As an example of the kind of project that we hope could be more efficient, we took the Maths in Context trial funded by the Education Endowment Foundation (EEF). This was a cutting-edge trial conducted to high standards that aimed to improve pupils’ maths capabilities (measured at GCSE level), with teaching interventions targeted at year 10s across 118 schools.

Part of the reason 118 schools were required was because the available data didn’t allow pupils’ GCSE scores to be linked to individual teachers. This meant that all teachers in any one school would have to receive the intervention, or not, because the school was the only level at which GCSE results could be linked to getting the intervention. The project enlisted 118 schools – with 50% getting the intervention – as this was enough to observe a ‘minimum detectable effect’ of 0.2 standard deviations in the GCSE Maths results – a smallish, but important effect.

For our exploration of how TED could have helped reduce the size of the trial, we mirrored the Maths in Context trial using our TED data, which is made up of data from our founding multi-academy trusts. There were several important differences in how we simulated the trial with the TED: we used a much smaller group of schools, we used an end of year 7 Maths test instead of GCSE maths, and, since the TED allows us to anonymously link pupils’ test scores with individual teachers, we simulated allocating the intervention individual teachers within each school – instead of at the whole-school level as in the EEF’s trial.

We also included different types of subject data alongside Maths to understand the impact including this data might have on the results. We tested four scenarios:

1. KS2 Maths and Reading scores and demographics
2. The above plus other subjects like French, History, and English
3. The above plus Maths scores from the Autumn term
4. All the above combined

To make the comparison with Maths in Context, we fixed our desired minimum detectable effect at 0.2 standard deviations and then ran our simulation to see how many schools would be required so that an effect of this size would be reliably detected with standard statistical tests.

What we found

The table below shows our results.

The first line shows the number of schools in the Maths in Context trial, 118. The following four lines show the number of schools required to observe our desired minimum detectable effect of 0.2 for each of the four scenarios set out above. The results show that by simply moving to the within-school design and using a year 7 test instead of GCSE, the number of schools required to observe the effect of an intervention drops to 44. Including additional data points reduced the number of schools even further, down to 42.

In short, the trial indicates that a within-school teacher-randomised trial could reduce the number of schools required for similar interventions by over 70%. 

Considerations

This is a very simple simulation and there are other things to consider, of course, if we were actually running this trial with the TED. For example, we might still want to use GCSEs as the outcome measure because they are such high-stakes tests, and as our simulation shows, adding concurrent results from other subjects (e.g., GCSEs) to the trial could reduce the number of schools needed.

There might also be good arguments for randomising at the whole school level because of how the programme is delivered; some interventions are ‘whole school’ and cannot be delivered to individual teachers in the same school.

That said, there are also other advantages to the simulated design beyond reducing the number of schools. For example, focusing on a relatively low-stakes year group may make it easier to recruit schools to future trials, and randomising teachers within schools means all schools would get to experience some level of intervention.

Further research could look at how outcomes in other year groups and subjects might be affected by using this dataset.

Conclusion

Our findings have exciting implications for researchers, teacher educators, and schools, highlighting the potential for significant cost and resource savings in future evaluations of new approaches to teaching.

By using schools’ data intelligently and efficiently while preserving the anonymity of teachers, there is theoretically no limit to what we might be able to test in pursuit of the very best educational outcomes for future generations of children.

Further information on the teacher-education dataset is available on the NIoT website, where you can also subscribe to receive timely updates on all of our research programmes.