Integrating English: Evaluation Report

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The trial randomised whole schools rather than classes or students, which is stronger than and automatically satisfies class-level randomisation.
    • "The impact evaluation design was a two-arm, school-level clustered randomised controlled trial (CRCT). School level randomisation was used as it is more practical for recruitment and implementation. It also eliminates the risk of overspill, which is crucial for ensuring robustness." (p. 14)
    • Relevant Quotes: 1) "This was a randomised controlled trial involving 91 schools. Schools were randomly allocated to either receive the Integrating English programme, or to be in a 'business as usual' control." (p. 5) 2) "The impact evaluation design was a two-arm, school- level clustered randomised controlled trial (CRCT). School level randomisation was used as it is more practical for recruitment and implementation. It also eliminates the risk of overspill, which is crucial for ensuring robustness." (p. 14) 3) "The MinimPy programme was used to allocate schools into the intervention and control groups using minimisation with a random start. Five school-level variables were used to ensure a good baseline balance between the two groups." (p. 16) Detailed Analysis: The ERCT 'C' criterion requires randomisation at the class level or stronger, with the whole school level explicitly accepted as satisfying (indeed exceeding) this requirement. The paper is explicit and consistent throughout that the unit of randomisation was the whole school, not individual classes or students within a school. Ninety-one schools were recruited and then randomly allocated using a minimisation algorithm to intervention or control status, with entire year groups (all Y5, then all Y6, classes) within intervention schools receiving the programme. This is a considerably stronger design than class-level randomisation and removes any risk of within-school contamination between conditions. Since school-level randomisation automatically satisfies the weaker class-level criterion under the ERCT standard, criterion C is met.
  • E

    Exam-based Assessment

    • The primary outcome used genuine past national KS2 Writing test papers externally marked, and the secondary outcomes (Reading, GPS) were drawn from compulsory national KS2 testing, both standardised measures.
    • "After considering the options, it was agreed to use KS2 Writing past papers from 2012, marked by an organisation contracted by the evaluation team." (p. 15)
    • Relevant Quotes: 1) "The primary outcome measure was writing, assessed through a test administered by the evaluators (using KS2 past papers from 2012) for all pupils in the target year group (starting in Y5, ending in Y6)." (p. 15) 2) "After considering the options, it was agreed to use KS2 Writing past papers from 2012, marked by an organisation contracted by the evaluation team." (p. 15) 3) "The markers were all blind to study allocation." (p. 15) 4) "The secondary outcomes were attainment in KS2 Grammar, Punctuation and Spelling (GPS) and KS2 Reading. Raw scores, as available from the National Pupil Database (NPD), were used for the impact analyses. These secondary outcome tests were marked externally as part of compulsory national KS2 testing." (p. 16) Detailed Analysis: The primary outcome was not a bespoke instrument invented to flatter the intervention; it used genuine past national curriculum KS2 Writing test papers (from 2012), marked externally by a contracted organisation blind to allocation, because the current (post-2016) KS2 writing marking scheme produced only categorical rather than scale scores unsuitable for the trial's statistical design. The two secondary outcomes, KS2 Reading and KS2 Grammar, Punctuation and Spelling, are drawn directly from the compulsory, nationally standardised Key Stage 2 testing regime via the National Pupil Database. Because all three outcome measures are either genuine past national standardised assessments or the current compulsory national assessments, rather than instruments custom-built and validated only by the evaluators for this study, criterion E is met.
  • T

    Term Duration

    • Outcomes were measured around 20 months after the intervention began, far exceeding the minimum one-term requirement (and this is superseded by the Year Duration criterion also being met).
    • "The trial took place between October 2016 and July 2018." (p. 5)
    • Relevant Quotes: 1) "The trial took place between October 2016 and July 2018." (p. 5) 2) "The intervention spanned two years." (p. 5) 3) "Baseline tests were taken between 31 October and 25 November 2016 and administered by the evaluation team." (p. 15) 4) "The primary outcome assessment, which was administered in schools during June and July 2018 ..." (p. 12) Detailed Analysis: Y5 teachers began LiLAC training in spring term 2016/2017, taught pupils in summer term 2016/2017, and Y6 training and teaching continued through to spring term 2017/2018, with the primary outcome writing assessment administered in summer 2018 -- roughly 20 months after the intervention's classroom teaching first began, and well over a year from first training. This vastly exceeds the minimum one full academic term required by criterion T. Because a full academic year (and more) of tracking is documented, and the stronger Year Duration criterion (Y) is independently met below, the weaker Term Duration criterion is automatically satisfied. Criterion T is met.
  • D

    Documented Control Group

    • The control group is documented in detail, with a full comparison table of baseline demographic and attainment characteristics for both arms and a description of control conditions.
    • "Table 7 below compares the intervention and control group samples at baseline (91 schools, 4,762 pupils) and in the ITT analyses for the primary KS2 writing outcome. This shows good balance at both baseline and outcome in terms of hub area, OFSTED, number of classes per school, school- level KS1 and KS2 attainment, %EAL, %FSM, Fluency in English, and pupil-level GL baseline test score." (p. 24)
    • Relevant Quotes: 1) "The control condition was 'business as usual'. Control schools were given £200 for taking part." (p. 14) 2) "Table 7 below compares the intervention and control group samples at baseline (91 schools, 4,762 pupils) and in the ITT analyses for the primary KS2 writing outcome. This shows good balance at both baseline and outcome in terms of hub area, OFSTED, number of classes per school, school- level KS1 and KS2 attainment, %EAL, %FSM, Fluency in English, and pupil-level GL baseline test score." (p. 24) 3) "We have no evidence that teachers or pupils in control schools received the Integrating English programme, and so assume that all control schools can be considered as compliant with the control condition." (p. 17) Detailed Analysis: The control condition is explicitly described (business as usual, with a modest £200 incentive payment and no Integrating English training or materials), and Table 7 provides an extensive, itemised comparison of intervention versus control schools and pupils across hub area, Ofsted grade, class numbers, prior KS1/KS2 attainment, %EAL, %FSM, English fluency-scale distribution, and baseline test scores, both at randomisation and at final analysis. This is a thorough, quantified documentation of the control group's composition and comparability, going well beyond a bare assertion that a control group existed. Criterion D is met because the control group's size, demographic and attainment characteristics, and treatment conditions are clearly and quantitatively documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was conducted at the level of whole schools using a formal minimisation procedure, directly satisfying the school-level RCT requirement.
    • "The impact evaluation design was a two-arm, school-level clustered randomised controlled trial (CRCT)." (p. 14)
    • Relevant Quotes: 1) "The impact evaluation design was a two-arm, school- level clustered randomised controlled trial (CRCT). School level randomisation was used as it is more practical for recruitment and implementation." (p. 14) 2) "The MinimPy programme was used to allocate schools into the intervention and control groups using minimisation with a random start. Five school-level variables were used to ensure a good baseline balance between the two groups." (p. 16) 3) "Randomised (school n= 91; pupil n=4762) ... Intervention (school n=46; pupil n=2483) ... Control (school n=45; pupil n=2279)" (Figure 2, p. 23) Detailed Analysis: The trial explicitly randomised at the level of the whole school (91 schools total, 46 to intervention and 45 to control), using a minimisation algorithm balancing five school-level variables (mean KS2 attainment, %EAL, EAL fluency distribution, number of Y5 classes, and geographical hub). This is a clear, well-documented instance of school-level cluster randomisation rather than class- or student-level allocation. Criterion S is met because entire schools, not merely classes or students within schools, were the randomised unit.
  • I

    Independent Conduct

    • The trial was independently evaluated by Sheffield Hallam University with blind, externally-contracted test markers, separate from the Enfield LA team that designed and delivered the intervention.
    • "This evaluation was led by the Sheffield Institute for Education and funded by the EEF, the Bell Foundation, and Unbound Philanthropy." (p. 11)
    • Relevant Quotes: 1) "The programme was developed and delivered by Enfield LA School Improvement Team, and was funded by the EEF, Unbound Philanthropy, and The Bell Foundation." (p. 5) 2) "This evaluation was led by the Sheffield Institute for Education and funded by the EEF, the Bell Foundation, and Unbound Philanthropy." (p. 11) 3) "The markers were all blind to study allocation." (p. 15) 4) "Evaluation team (all Sheffield Hallam University) ... Delivery team ... Project manager: Sharon Davies ... Project lead and LiLAC tutor: Michelle Stanley ... Enfield Council" (p. 13) Detailed Analysis: There is a clear organisational separation between the body that designed and delivered the Integrating English programme (Enfield LA School Improvement Team, delivering a training course, LiLAC, itself owned by the Government of South Australia and developed independently of Enfield and the evaluators) and the body that designed, ran, and analysed the evaluation (the Sheffield Institute of Education at Sheffield Hallam University). Outcome test papers were marked by external contracted organisations (GL Assessment for baseline, another contractor for the primary outcome) who were blind to treatment allocation, further reducing measurement bias. One limitation noted is that "Analysis was not conducted blinded to randomisation" (p. 16) because the main analyst was also project manager, but this concerns analytic blinding rather than independence of conduct from the intervention's designers. Criterion I is met because data collection and analysis were conducted by an evaluation team organisationally independent of, and with no role in designing, the Integrating English/LiLAC intervention, with blinded external markers used for outcome testing.
  • Y

    Year Duration

    • Pupils were tracked from the start of the intervention in Y5 through to final outcome testing roughly 20 months later at the end of Y6, comfortably exceeding 75% of an academic year.
    • "The intervention spanned two years." (p. 5)
    • Relevant Quotes: 1) "The intervention spanned two years. In the first year, Year 5 teachers were trained for five days in the spring term before producing schemes of work ... These were delivered to pupils in the summer term of Year 5. During this summer term and the following autumn term, Year 6 teachers were then trained ... before designing and delivering schemes of work to the same students, now in Year 6." (p. 5) 2) "The trial took place between October 2016 and July 2018." (p. 5) 3) "Baseline tests were taken between 31 October and 25 November 2016" (p. 15); "the primary outcome assessment, which was administered in schools during June and July 2018" (p. 12) Detailed Analysis: Pupils entered the trial as Y5 students in autumn 2016, received Integrating English teaching in the summer term of Y5 (2017) and again in the spring term of Y6 (2018), and had their primary and secondary outcomes measured in summer 2018. This gives a tracking period of roughly 20 months from first intervention exposure to final outcome measurement, and well over the full academic year from the November 2016 baseline to the summer 2018 outcome testing -- substantially more than the 75% of an academic year required. Criterion Y is met, and this in turn confirms the weaker Term Duration criterion (T) is also satisfied.
  • B

    Balanced Control Group

    • The additional resource in this trial was teacher professional development delivered outside pupil instructional time; pupils in both arms received the same amount of normal classroom time, with the control condition appropriately remaining business-as-usual as this is precisely the treatment being tested.
    • "The intervention is delivered in normal classroom time. It is expected that teachers incorporate the principles learnt in the training into their everyday teaching. There are no ongoing costs associated with delivery." (p. 30)
    • Relevant Quotes: 1) "The intervention is delivered in normal classroom time. It is expected that teachers incorporate the principles learnt in the training into their everyday teaching. There are no ongoing costs associated with delivery." (p. 30) 2) "The control condition was 'business as usual'. Control schools were given £200 for taking part." (p. 14) 3) "The Integrating English programme is designed to improve the subject literacy and language pedagogy of teachers, and in turn improve the linguistic proficiency and communicative ability for all students." (p. 8) Detailed Analysis: Applying the decision procedure for criterion B: the extra resource here is teacher continuing professional development (five days of LiLAC training per year group, approximately 50 hours total plus supported scheme-of-work preparation), which is delivered to teachers outside pupil class time; teaching to pupils itself occurs within "normal classroom time" for both intervention and control schools, with no additional instructional hours, materials budget, or class time given to intervention pupils relative to control pupils. The additional resource that is not matched (teacher training) is integral to, and is precisely, the treatment variable under investigation -- the trial is testing whether a professional development programme changes teaching practice and pupil outcomes, not whether extra pupil-facing time or materials improve outcomes. Control schools were kept on a "business as usual" footing and additionally received a modest £200 payment, consistent with the standard EEF practice of incentivising participation without providing a competing intervention. Because the differential resource (teacher training) is the explicit subject of the treatment being tested and pupils in both arms received equivalent classroom time and materials, criterion B is met.

• Level 3 Criteria

  • R

    Reproduced

    • This is described as the first large-scale evaluation of this programme, with no independent replication reported or evident in the paper, and none was located through internet research.
    • "The current project, Integrating English, is consequently the first large-scale (multi-site, multi-region) evaluation of a general implementation of genre pedagogy outside of Australia." (p. 7)
    • Relevant Quotes: 1) "The current project, Integrating English, is consequently the first large-scale (multi-site, multi- region) evaluation of a general implementation of genre pedagogy outside of Australia." (p. 7) 2) "evaluations of genre pedagogy, particularly the LiLAC (Language in Learning Across the Curriculum) programme (Custance, Dare and Polias, 2012) at the centre of the current evaluation, have thus far typically been case- based (Pomagalska, 2019), have focused on either writing or reading (Rose and Acevedo, 2006), or have used measures that limit reliable evaluation (Coffin, Acevedo and Lövstedt, 2013)." (p. 7) Detailed Analysis: The paper is explicit that no prior large-scale, RCT-based evaluation of this specific programme (Integrating English, built on LiLAC) exists; earlier work on LiLAC/genre pedagogy is described as small-scale, case-based, or methodologically limited, and is not an independent replication of this trial's design, population, or outcome measures. The report itself represents the first rigorous evaluation rather than a replication of an earlier RCT. Internet Search for Independent Replication: A search was conducted for independent replications of this specific trial (a school-level clustered RCT of the LiLAC-based Integrating English programme in English primary schools) by other research teams. No independent replication was found. Subsequent related work by the same evaluation team (e.g. Coldwell et al., 2024, "Learning from failure: A context-informed perspective on RCTs", British Educational Research Journal) discusses this trial's implementation failure as a case study but does not constitute a replication by a different team. Other LiLAC/ genre-pedagogy evaluations located in searches (e.g. Australian case studies such as Pomagalska, 2019) are not replications of this specific CRCT's design, population, or outcome measures. Criterion R is not met because there is no evidence, either within this paper or from further internet research, of an independent replication of this specific study by a different research team.
  • A

    All-subject Exams

    • Only English-domain outcomes (writing, reading, grammar/ punctuation/spelling) were assessed; no other core subjects such as mathematics or science were measured.
    • "The Impact Evaluation addressed the following questions: 1. What is the impact of the intervention on the language ability of Y6 pupils, as measured by a writing test (KS2 past paper) as primary outcome, with KS2 Reading and KS2 English Grammar, Punctuation and Spelling (GPS) as secondary outcomes." (p. 11)
    • Relevant Quotes: 1) "The Impact Evaluation addressed the following questions: 1. What is the impact of the intervention on the language ability of Y6 pupils, as measured by a writing test (KS2 past paper) as primary outcome, with KS2 Reading and KS2 English Grammar, Punctuation and Spelling (GPS) as secondary outcomes." (p. 11) 2) "Table 2: Trial design summary ... Primary outcome variable: Writing ... Secondary outcome(s) variable(s): KS2 Reading; KS2 Grammar, Punctuation and Spelling (GPS)" (p. 14) Detailed Analysis: Although Integrating English is explicitly designed as a cross-curricular intervention (intended to affect language use in science, history, mathematics and other subjects, as described in the Introduction), the impact evaluation's primary and both secondary outcomes are all narrowly within the English/literacy domain (writing, reading, grammar- punctuation-spelling). No standardised assessment of mathematics, science, or other core curriculum subjects was administered to check for possible negative (or positive) spillover effects outside English. The paper's own Limitations section acknowledges the difficulty of capturing the cross-curricular literacy focus of the intervention with a single writing test, but this does not substitute for assessing other core subjects. Criterion A is not met because outcome measurement was confined to the English/literacy domain and did not extend to other core subjects taught in the school.
  • G

    Graduation Tracking

    • Pupils were followed from Y5 through to the completion of Y6 and the KS2 assessment, which marks the end of primary schooling in England before transition to secondary school; no follow-up publication tracking this cohort further was found.
    • "The primary outcome measure was writing, assessed through a test administered by the evaluators (using KS2 past papers from 2012) for all pupils in the target year group (starting in Y5, ending in Y6)." (p. 15)
    • Relevant Quotes: 1) "The primary outcome measure was writing, assessed through a test administered by the evaluators (using KS2 past papers from 2012) for all pupils in the target year group (starting in Y5, ending in Y6)." (p. 15) 2) "Spring term 2017/2018 ... 6. implement scheme of work ... teaching experienced by pupils now in Y6" (Table 1, p. 10) 3) "Summer 2018 ... Key Stage 2 testing ... Outcome assessment administration and marking" (Table 5, p. 21) Detailed Analysis: The cohort under study was tracked continuously from Y5 (the point at which the intervention began) through to the completion of Y6, when the KS2 (Key Stage 2) assessments were taken. In the English education system, KS2 testing at the end of Y6 marks the completion of, and formal exit point from, primary education, immediately before pupils transition to secondary school -- functionally analogous to a "graduation" assessment for that phase of schooling. This mirrors the criterion's stated aim of tracking students "until graduation" for the relevant educational stage "without needing to track them after leaving school." No further follow-up beyond KS2/end of primary school was conducted or was necessary to satisfy this criterion, since primary schooling was itself completed within the study's tracking window. Internet Search for Follow-up Publications: A search was conducted for subsequent publications by the same author team (Culliney, Moore, Coldwell, Demack) that might track this cohort further, for example into secondary school KS4/GCSE outcomes. No such follow-up outcome-tracking study was found. A later methodological paper by the evaluation team (Coldwell et al., 2024, British Educational Research Journal) reflects on this trial's implementation but does not report further pupil outcome data beyond the KS2 measures already covered here. Criterion G is met because pupils were followed through to the natural completion ("graduation") point of the primary- school stage in which the intervention was delivered, and no further tracking evidence was found to reconsider this.
  • P

    Pre-Registered

    • The trial has a public ISRCTN registry entry, but neither the paper nor further internet research could confirm that registration occurred before data collection began.
    • "The trial is publicly registered under International Standard Randomised Controlled Trial Number (ISRCTN) ISRCTN44415239: http://www.isrctn.com/ISRCTN44415239" (p. 11)
    • Relevant Quotes: 1) "The trial is publicly registered under International Standard Randomised Controlled Trial Number (ISRCTN) ISRCTN44415239: http://www.isrctn.com/ISRCTN44415239" (p. 11) 2) "Protocol: .../Integrating_English_Protocol_Amended_ 2017.11.07.pdf" and "SAP: .../Integrating_English_SAP_ 2017.11.07___FINAL.pdf" (p. 12) 3) "Baseline tests were taken between 31 October and 25 November 2016" (p. 15) Detailed Analysis: The trial does have a public ISRCTN registration and cites a protocol and statistical analysis plan, which is supportive of good pre-registration practice. However, the text in the report does not itself quote a registration date for the ISRCTN entry, and the only dated protocol documents referenced ("Protocol ... Amended 2017.11.07" and "SAP ... 2017.11.07") post-date the November 2016 baseline data collection by roughly a year, raising the possibility that these particular protocol versions were amendments made after data collection had already begun (the SAP is explicitly described elsewhere as "agreed" in November 2017, after baseline testing in 2016). Internet Search for Pre-registration Timing: An attempt was made to check the ISRCTN44415239 registry record directly to establish the original registration date relative to the October/November 2016 baseline testing start. The ISRCTN registry's individual trial pages require JavaScript/cookies and did not return retrievable registration-date content via automated web search or fetch, so the original registration date could not be independently confirmed or refuted from available sources. Without an explicit quote or independently retrieved date confirming that the original pre-registration predates the start of data collection, this criterion cannot be confirmed as met. Criterion P is not met because neither the paper nor further internet research establishes that registration of the full study protocol occurred before data collection began.