PiloTED: laying the foundations for TED

Project overview

Schools in England are facing significant retention and recruitment challenges, making it critical to invest in the effective development of existing teachers. However, policymakers and school leaders currently lack a reliable way to measure teacher impact, which is essential for understanding which professional development pathways are most effective.

The growth of MATs provides a unique opportunity: within some trusts, thousands of pupils across multiple schools sit the same assessments at the same time, often following a shared curriculum. This research project, led by Professor Rob Coe at Evidence Based Education and Dr Raj Chande at NIoT, explored the potential of using schools’ anonymised internal assessment data to estimate teacher impact while maintaining strict anonymity for teachers, schools, and pupils.

Specifically, the project aimed to:

• Establish acceptability: by assessing whether teachers and school leaders find the use of anonymised assessment data to estimate teacher impact acceptable and appropriate.
• Ensure feasibility: to determine the practical viability of collecting and analysing internal school assessment data while maintaining robust anonymisation and reducing school burden.
• Evaluate quality: Determine whether the assessment data is of sufficient quality to facilitate this type of research
• Evaluate Validity: Investigate whether our statistical approach to estimating teacher impact is valid and reliable.

Methods

To evaluate the acceptability and feasibility of using anonymised pupil assessment data, we conducted online focus groups with teachers and leaders in our four partner MATs. The quality of the data was then assessed by examining (1) the range of scores achieved in each assessment, (2) how closely the difficulty level of the assessments matched the ability of the pupils taking them, and (3) correlations between assessments taken in the same and different subjects. Finally, several value-added models were developed to measure teacher impact using the anonymised pupil assessment data.

Summary of findings

We found that teachers and leaders generally supported the use of anonymised assessment data to evaluate teacher impact so that we might identify effective teacher development practices, provided that the data remains anonymous and is not used for performance management.

We also found that while collecting and matching assessment data within schools is complex and demanding, it is feasible with concerted effort, collaboration with MAT data managers, and proper data management practices. Assessments were found to be of sufficient quality for our intended analyses, although some assessments were of better quality than others. For example, some assessments were too easy as too many pupils achieved the top scores, and some were too hard as many pupils got the lowest scores. However, while a small minority could be improved upon, most assessments were found to be suitable for estimating teacher impact. Finally, our value-added models produced reliable estimates of teacher impact, and not just statistical ‘noise’.

Project team

• Professor Robert Coe, Evidence Based Education & EEF
• Dr Ourania Ventista, TIDE Researcher
• Dr Raj Chande, Senior Research Associate, NIoT
• Shaun Dillon, Director of Data, NIoT
• Claire Maud, Research Fellow, NIoT
• Professor Stuart Kime, Evidence Based Education

Click here to return to the TED overview