EEDI 2024 IMPACT REPORT: A study to evaluate the effectiveness of Eedi on raising attainment in mathematics at KS3 (Year 7)

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Whole schools were randomly assigned to intervention and control, exceeding the class-level randomisation requirement.
    • This study was designed as a randomised controlled trial with whole schools (Year 7) assigned at random to either intervention (exposure to Eedi) or control conditions (non-exposure to Eedi) on a 1:1 basis.
    • Relevant Quotes: 1) "This study was designed as a randomised controlled trial with whole schools (Year 7) assigned at random to either intervention (exposure to Eedi) or control conditions (non-exposure to Eedi) on a 1:1 basis." (p. 19) 2) "Randomisation was carried out by the WhatWorked team who were ‘blind’ as to the identity of schools." (p. 19) Detailed Analysis: The unit of randomisation is the school ("whole schools ... assigned at random"), which is stronger than the minimum class-level requirement for criterion C. The randomisation process is described and was conducted by the evaluation team. Criterion C is met because whole schools were randomly assigned to intervention and control conditions.
  • E

    Exam-based Assessment

    • Outcomes were measured using the standardised NWEA MAP Growth assessment rather than a custom test.
    • The primary outcome for this study was attainment in KS3 Year 7 mathematics as defined by the NWEA MAP Growth Standardised Assessment.
    • Relevant Quotes: 1) "The primary outcome for this study was attainment in KS3 Year 7 mathematics as defined by the NWEA MAP Growth Standardised Assessment." (p. 21) 2) "The online assessments are completed in the Autumn, Spring and Summer term and are standardised to US measures to allow for international comparisons for the grant funders." (p. 21) Detailed Analysis: The study used NWEA MAP Growth, described as a "Standardised Assessment" and "standardised to US measures", as the primary outcome. This is not a bespoke researcher-made test, so it satisfies the ERCT requirement for an external standardised exam-based assessment. Criterion E is met because the primary outcome measure was the standardised NWEA MAP Growth assessment.
  • T

    Term Duration

    • The study followed students from Autumn 2023 through the July 2024 post-assessment, exceeding one full term.
    • The Eedi programme was used in intervention schools from Autumn 2023 to June 2024 to supplement classroom teaching, with the control schools not using the programme.
    • Relevant Quotes: 1) "The Eedi programme was used in intervention schools from Autumn 2023 to June 2024 to supplement classroom teaching, with the control schools not using the programme." (p. 3) 2) "All participating Year 7 pupils completed the 45-minute online pre-test under examination conditions in the initial assessment window in September 2023, mid-point assessment in February 2024 and post-assessment in July 2024." (p. 21) Detailed Analysis: The intervention began in Autumn 2023 and outcomes were measured again at post-assessment in July 2024. This is well beyond the minimum requirement of one academic term (3-4 months) from intervention start to outcome measurement. Criterion T is met because the time from intervention start to the final outcome measurement spans far longer than one term.
  • D

    Documented Control Group

    • The report documents the control group definition, baseline school characteristics, and control schools business as usual practices.
    • Schools were only recruited if they did not currently use Eedi in Year 7 and if they agreed to participate and withhold the release of Eedi to Year 7 if they were selected as the control.
    • Relevant Quotes: 1) "Schools were only recruited if they did not currently use Eedi in Year 7 and if they agreed to participate and withhold the release of Eedi to Year 7 if they were selected as the control." (p. 20) 2) "Each school was allocated an anonymous ID and Eedi used the DfE performance tables and a WhatWorked School look up to record the current progress 8 score, attainment 8 score, percentage of schools eligible for Pupil Premium (PP), percentage of pupils with English not as their first language (EAL) and school type." (p. 19) 3) "The process evaluation also sheds light on the ‘business as usual’ practices in control schools, which were similar to intervention schools in terms of homework frequency and duration." (p. 49) Detailed Analysis: The control condition is explicitly defined as schools withholding access to Eedi for Year 7. The report documents baseline school characteristics using official sources (DfE performance tables) and identifies the main features of "business as usual" in control schools. This is sufficient documentation to understand who the control group is and what they did. Criterion D is met because the control group is clearly described and baseline characteristics and practices are documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Twenty schools were randomised at the school level, satisfying the school-level RCT requirement.
    • This impact evaluation included 20 schools, with randomisation at school level.
    • Relevant Quotes: 1) "This impact evaluation included 20 schools, with randomisation at school level." (p. 3) 2) "This study was designed as a randomised controlled trial with whole schools (Year 7) assigned at random to either intervention (exposure to Eedi) or control conditions (non-exposure to Eedi) on a 1:1 basis." (p. 19) Detailed Analysis: The unit of allocation is the school, and the report states that 20 schools took part. This satisfies ERCT criterion S, which requires school-level randomisation rather than only within-school class-level allocation. Criterion S is met because schools (not just classes) were randomly assigned to conditions.
  • I

    Independent Conduct

    • The report separates the developer from the evaluator, with WhatWorked conducting randomisation and evaluation activities.
    • Eedi are an education technology company (developers) and WhatWorked are a research evidence company (evaluators).
    • Relevant Quotes: 1) "Randomisation was carried out by the WhatWorked team who were ‘blind’ as to the identity of schools." (p. 19) 2) "The Eedi project is being delivered by Eedi in partnership with WhatWorked Education. Eedi are an education technology company (developers) and WhatWorked are a research evidence company (evaluators)." (p. 67) 3) "The project is funded by Eedi with the aim of developing an evidence base for the impact of their programme in schools." (p. 67) Detailed Analysis: The report distinguishes the developer (Eedi) from the evaluator (WhatWorked Education). Randomisation was conducted by the WhatWorked team and they report being blind to school identity during randomisation. Although the project is funded by Eedi, the evaluation is described as being carried out by a separate research evidence company. Criterion I is met because the evaluation and randomisation were carried out by an organisation separate from the intervention developer.
  • Y

    Year Duration

    • Outcomes were tracked from September 2023 to July 2024, covering an academic year.
    • All participating Year 7 pupils completed the 45-minute online pre-test under examination conditions in the initial assessment window in September 2023, mid-point assessment in February 2024 and post-assessment in July 2024.
    • Relevant Quotes: 1) "The Eedi programme was used in intervention schools from Autumn 2023 to June 2024 to supplement classroom teaching, with the control schools not using the programme." (p. 3) 2) "All participating Year 7 pupils completed the 45-minute online pre-test under examination conditions in the initial assessment window in September 2023, mid-point assessment in February 2024 and post-assessment in July 2024." (p. 21) Detailed Analysis: The intervention covered the 2023-2024 academic year and outcomes were tracked from September 2023 through July 2024, which is approximately a full school year. This satisfies the ERCT year-duration requirement. Criterion Y is met because the study tracked outcomes across a full academic year.
  • B

    Balanced Control Group

    • The intervention did not add unbalanced time or resources: homework frequency and duration were similar and controls used other platforms.
    • Reported duration of maths homework was similar across intervention and control schools, with responding teachers in both groups reporting that maths homework should take students approximately 30 minutes to complete (intervention n=34, control n=35)
    • Relevant Quotes: 1) "Reported frequency of maths homework was similar across intervention and control schools, with responding teachers in both groups reporting that maths homework is typically set by class teachers once a week (intervention n=56, control n=50)." (p. 52) 2) "Reported duration of maths homework was similar across intervention and control schools, with responding teachers in both groups reporting that maths homework should take students approximately 30 minutes to complete (intervention n=34, control n=35)" (p. 53) 3) "Control schools, on the other hand, displayed a broader usage of various online platforms throughout the trial. The most commonly used platforms in control schools were MathsWatch and Sparx, both of which are considered robust tools for addressing misconceptions and supporting students through targeted feedback and practice." (p. 54) Detailed Analysis: The key resource inputs relevant to this intervention are time and effort spent on maths homework and access to an online homework platform. Teacher survey data indicates similar homework frequency (once per week) and duration (about 30 minutes) in both groups. Control schools also used other well-known online maths platforms, suggesting that the intervention did not add materially more time or resources beyond business as usual. Criterion B is met because homework time expectations were similar and control schools used comparable online platforms.

• Level 3 Criteria

  • R

    Reproduced

    • No independent peer-reviewed replication of this specific school-level RCT was found.
    • This study follows on from an initial pilot in Year 1 for a pilot impact study to test the effectiveness of Eedi in KS3 mathematics in UK schools (Harrison, Brown & Higgins, 2023).
    • Relevant Quotes: 1) "This study follows on from an initial pilot in Year 1 for a pilot impact study to test the effectiveness of Eedi in KS3 mathematics in UK schools (Harrison, Brown & Higgins, 2023)." (p. 17) Detailed Analysis: The report cites a prior pilot study by the same author group, but it does not cite any independent replication conducted by a different research team and published in a peer-reviewed journal. A search of publicly available sources did not identify a peer-reviewed, independent reproduction of this specific 2023-2024 school-level RCT. Criterion R is not met because no independent peer-reviewed replication of this study could be located.
  • A

    All-subject Exams

    • Only mathematics attainment was assessed, not standardised outcomes in all core subjects.
    • The primary outcome for this study was attainment in KS3 Year 7 mathematics as defined by the NWEA MAP Growth Standardised Assessment.
    • Relevant Quotes: 1) "The primary outcome for this study was attainment in KS3 Year 7 mathematics as defined by the NWEA MAP Growth Standardised Assessment." (p. 21) Detailed Analysis: The study reports outcomes for mathematics only. There is no evidence in the report that standardised assessments were also collected for other core subjects (for example, reading/language arts or science) to test for cross-subject trade-offs. Criterion A is not met because only mathematics attainment was assessed.
  • G

    Graduation Tracking

    • Follow-up stops in July 2024 for the Year 7 cohort, with no graduation tracking reported.
    • All participating Year 7 pupils completed the 45-minute online pre-test under examination conditions in the initial assessment window in September 2023, mid-point assessment in February 2024 and post-assessment in July 2024.
    • Relevant Quotes: 1) "Schools were allocated to intervention and control conditions using random allocation on a 1:1 basis for access to Eedi for the Year 7 cohort for the academic year 2023-24." (p. 19) 2) "All participating Year 7 pupils completed the 45-minute online pre-test under examination conditions in the initial assessment window in September 2023, mid-point assessment in February 2024 and post-assessment in July 2024." (p. 21) Detailed Analysis: The reported follow-up ends with the post-assessment in July 2024 for the Year 7 cohort. The report does not describe any subsequent tracking of these pupils through to graduation (for example, GCSE completion in Year 11), and no later follow-up publication by the same team reporting graduation outcomes was located. Criterion G is not met because students were not tracked until graduation.
  • P

    Pre-Registered

    • The report provides no trial registration or protocol ID, and no public pre-registration could be verified.
    • Relevant Quotes: None found. Detailed Analysis: The report does not mention pre-registration, trial registration, or a public protocol identifier (for example OSF, ISRCTN, or a registry link) dated before the intervention began. A search of major registries and the open web did not locate a public pre-registered protocol for this study. Criterion P is not met because no pre-registered protocol could be verified.