Teaching Grammar and Writing: A Randomised Controlled Trial and Implementation and Process Evaluation of Englicious

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was carried out at the school/teacher level (a stratified school-level cluster trial), which meets or exceeds the class-level requirement.
    • "Randomisation was carried out at the school/teacher- level (only one teacher was recruited within each school, hence these two are indistinguishable for research design purposes)."
    • Relevant Quotes: 1) "Randomisation was carried out at the school/ teacher-level (only one teacher was recruited within each school, hence these two are indistinguishable for research design purposes)." (p. 124) 2) "Preliminary power calculations for what was a stratified school-level cluster trial randomisation indicated that for the target sample of 60 schools..." (p. 124) 3) "Overall: 70 schools (34 treatment; 36 control)" (p. 125) 4) "The schools/teachers in the sample were randomly allocated either to the intervention or the control group." (p. 127) Detailed Analysis: Criterion C requires an RCT with randomisation at the class level or stronger. The study is a cluster RCT in which whole schools (each with a single Year 2 class/ teacher) were the unit of randomisation. Because only one class per school participated, randomising the school is equivalent to randomising the class, and it is at least as strong as class-level assignment, preventing within-class contamination. The ERCT standard states that when a study is a school-level RCT, the weaker class-level criterion is considered met. Criterion C is met because randomisation was performed at the school/teacher (cluster) level, which satisfies or exceeds the class-level requirement.
  • E

    Exam-based Assessment

    • The primary outcome was the standardised Progress in English (PiE) writing test from GL Assessment, a widely recognised standardised assessment.
    • "Progress in English (PiE) test: the primary outcome measure. A standardised measure to assess pupils' writing through a narrative writing test."
    • Relevant Quotes: 1) "Progress in English (PiE) test: the primary outcome measure. A standardised measure to assess pupils' writing through a narrative writing test. We secured agreement from GL Assessment to use the element from their Progress in English (PiE) test that was focussed on writing." (p. 129) 2) "For the PiE measure the reliability was found to be greater than 0.9 (Cronbach's Alpha; Kirkup et al., 2006)." (p. 130) 3) "Sentence Generation Test (SG): the secondary outcome measure. A bespoke measure to assess grammar through pupils generating sentences from word prompts." (p. 129) Detailed Analysis: Criterion E requires a standardised, widely recognised exam-based assessment rather than a test custom-built for the study. The primary outcome measure was the Progress in English (PiE) test, a commercially published standardised writing assessment from GL Assessment (an established UK standardised-test publisher; the instrument and its marking criteria are documented by Kirkup et al., 2006, GL Assessment) with reported reliability greater than 0.9. The study used the writing element of this externally published instrument, not a study-built test. Although the secondary Sentence Generation measure is described as "bespoke", the criterion depends on the primary outcome, which relies on a recognised standardised instrument, so the criterion is satisfied. Criterion E is met because the primary outcome used the standardised, externally published Progress in English (PiE) writing test from GL Assessment.
  • T

    Term Duration

    • The intervention ran as ten weekly one-hour lessons with outcome measurement roughly three months after baseline, spanning approximately one academic term.
    • "With regard to the sentence generation task, as the same word pairs were used at baseline and immediate post intervention ... the tests were implemented approximately three months apart."
    • Relevant Quotes: 1) "The main focuses of the 10 one-hour weekly lessons in the intervention were as follows..." (p. 127) 2) "Pupils were tested twice on the writing outcome measures: 1. Baseline: prior to the start of the intervention; 2. Post-intervention: one to two weeks after the end of the intervention." (p. 130) 3) "...as the same word pairs were used at baseline and immediate post intervention it is possible that there could have been gains due to the children remembering the task although the tests were implemented approximately three months apart." (p. 151) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. The intervention was delivered as ten weekly one-hour lessons and post-intervention testing occurred one to two weeks after the intervention ended, with baseline and post tests "approximately three months apart". The interval from intervention start to outcome measurement therefore covers roughly one English academic term of weekly delivery. Criterion T is met because the interval from intervention start to outcome measurement spans approximately one full academic term.
  • D

    Documented Control Group

    • The business-as-usual control group is documented with demographic and baseline data in Tables 1 and 2 and a description of its conditions.
    • "Teachers in the business as usual control condition were asked to continue their usual practices for grammar teaching."
    • Relevant Quotes: 1) "Teachers in the business as usual control condition were asked to continue their usual practices for grammar teaching." (p. 128) 2) "Data needed for the primary analysis were collected for 1,246 pupils in total, with 637 in treatment and 609 in control." (p. 136) 3) "Tables 1 and 2 present the baseline characteristics of pupils and schools in the treatment and control groups." (p. 136) 4) "Table 1. Baseline characteristics of groups as randomised ... PIE Short/Baseline Score ... SG Baseline Score ... Female ... Ever FSM ... English as an additional language" (p. 139) Detailed Analysis: Criterion D requires clear documentation of the control group's composition, size, baseline performance, and conditions. The paper provides a dedicated control column in Tables 1 and 2 reporting sample sizes, gender, free-school-meal eligibility, English-as-an-additional-language proportions, and baseline PiE and Sentence Generation scores. It also states the control group followed business-as-usual grammar teaching, confirming no special treatment beyond normal schooling. Criterion D is met because the control group's characteristics, size, and business-as-usual condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The trial was an explicitly described school-level cluster RCT with 70 schools randomly allocated to intervention or control.
    • "Preliminary power calculations for what was a stratified school-level cluster trial randomisation..."
    • Relevant Quotes: 1) "Randomisation was carried out at the school/ teacher-level..." (p. 124) 2) "Preliminary power calculations for what was a stratified school-level cluster trial randomisation indicated that for the target sample of 60 schools..." (p. 124) 3) "Seventy schools and Year 2 teachers agreed to participate in the trial, met the eligibility criteria, and provided the necessary data for randomization to proceed." (p. 124) 4) "Overall: 70 schools (34 treatment; 36 control)" (p. 125) Detailed Analysis: Criterion S requires randomisation at the school level, where 'school' means the institution or unit implementing the intervention. The study explicitly describes a stratified school-level cluster trial in which 70 primary schools were randomly allocated to the Englicious intervention or the business-as-usual control. This is genuine school-level randomisation, not merely class- or student-level assignment. Criterion S is met because whole schools were the unit of random allocation in this school-level cluster RCT.
  • I

    Independent Conduct

    • The Englicious approach was designed by the linguists on the research team, who also delivered the training, ran the study and analysed the data, so it was not independently conducted.
    • "The Englicious teaching approach was designed by the linguists on the research team."
    • Relevant Quotes: 1) "The Englicious teaching approach was designed by the linguists on the research team." (p. 121) 2) "This training was led by the linguists on the research team who had developed Englicious." (p. 127) 3) "The intervention condition was explicitly designed, by the linguists on the team, and teachers, to meet the requirements of the grammar content of the national curriculum." (p. 150) 4) "All marking for the project was carried out by 23 markers ... markers were kept blinded from whether any given test they were allocated was treatment or control." (pp. 130-131) Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the designers of the intervention, to reduce bias in implementation, analysis, and reporting. Here the intervention (Englicious) was created by linguists (e.g., co-author Bas Aarts) who are members of the authoring research team; the same team delivered the teacher training, ran the trial, and carried out the statistical analysis. Although tests were marked by blinded recruited markers and administered by class teachers, the core design and analysis were performed by the intervention's developers, with no external, third-party evaluator overseeing the study. Criterion I is not met because the intervention designers and the study evaluators/analysts were the same research team, with no independent conduct.
  • Y

    Year Duration

    • Outcomes were measured only about three months after baseline, far short of 75% of an academic year.
    • "We planned for additional delayed tests but due to COVID-19 this was not possible, hence a deviation from the published protocol."
    • Relevant Quotes: 1) "The main focuses of the 10 one-hour weekly lessons in the intervention were as follows..." (p. 127) 2) "Post-intervention: one to two weeks after the end of the intervention." (p. 130) 3) "We planned for additional delayed tests but due to COVID-19 this was not possible, hence a deviation from the published protocol." (p. 130) 4) "...the tests were implemented approximately three months apart." (p. 151) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of a full academic year (roughly 7 or more months) after the intervention begins. This study delivered ten weekly lessons and measured outcomes one to two weeks after the intervention ended, with baseline and post tests only about three months apart. Planned delayed follow-up tests were abandoned due to COVID-19. The tracking interval is therefore roughly one term, well below the year-duration threshold. Criterion Y is not met because outcomes were measured only about three months after baseline, far short of 75% of an academic year.
  • B

    Balanced Control Group

    • The additional writing practice and grammar training are integral to the Englicious approach that is the explicit treatment variable, tested against a business-as-usual control.
    • "The intervention group had practice in writing, something that is not consistently done in regular teaching (Dockrell et al., 2015)."
    • Relevant Quotes: 1) "Teachers in the intervention group received one full day of in-person professional development explaining the rationale for the Englicious intervention and how it was to be implemented." (p. 127) 2) "In the final part of each lesson pupils were asked to carry out a short piece of writing using the grammar they had been taught, which was then shared and discussed in the class as a whole (20 minutes)." (p. 128) 3) "Teachers in the business as usual control condition were asked to continue their usual practices for grammar teaching." (p. 128) 4) "The intervention group had practice in writing, something that is not consistently done in regular teaching (Dockrell et al., 2015)." (p. 151) 5) "The hypothesis of the research was that the Englicious intervention that involved explicit grammar teaching combined with practising writing would improve six-year-old and seven-year-old pupils' sentence generation and narrative writing more than the varied teaching of grammar in 'business as usual' control classes." (pp. 123-124) Detailed Analysis: Criterion B asks whether the intervention adds time, budget, or materials not matched in the control, and if so whether those additional inputs are integral to the treatment being tested. The Englicious group received a one-day teacher professional development session and lessons that combined explicit grammar teaching with an embedded writing-practice element, whereas control teachers continued business-as-usual grammar teaching required by the same statutory national curriculum. Both groups were obliged to teach the same grammar content and delivered grammar within normal curriculum time; the distinguishing feature of the intervention is the specific Englicious pedagogy (grammar teaching directly linked to writing practice). The study's stated hypothesis makes this combined approach the explicit treatment variable being tested against a business-as-usual baseline, so the additional writing practice and the associated teacher training are integral to the intervention rather than separable confounds. Under the ERCT decision rule, when the added inputs are integral to the treatment being tested against business-as-usual, the criterion is satisfied. Criterion B is met because the additional writing practice and teacher training are integral components of the Englicious approach that constitutes the explicit treatment variable, compared against a business-as-usual control.

• Level 3 Criteria

  • R

    Reproduced

    • This is the first study of its kind with this age group and has not been independently replicated.
    • "The study reported in this paper was the first to adopt a combined RCT and Implementation and Process Evaluation (IPE) design on grammar teaching and writing with children aged six to seven."
    • Relevant Quotes: 1) "The study reported in this paper was the first to adopt a combined RCT and Implementation and Process Evaluation (IPE) design on grammar teaching and writing with children aged six to seven." (p. 123) 2) "...an approach that had not previously been subject to robust evaluation." (p. 121) 3) "...it is not one that has been found in previous research of grammar for writing approaches with children aged six to seven-years-old." (p. 150) Detailed Analysis: Criterion R requires independent replication of the specific study by a different research team, published in a peer-reviewed journal. The paper repeatedly emphasises that this is the first RCT of the Englicious approach and the first RCT of grammar/writing teaching with children aged six to seven. Prior cited studies (Myhill et al., 2011; Torgerson et al., 2014) evaluate different interventions or older pupils and are not replications of this study. An external check (paper references and available web sources; note the session's WebSearch quota was exhausted so only targeted URL fetches were possible) found no independent reproduction of this specific trial, and the paper itself frames the study as the first of its kind. Criterion R is not met because no independent replication of this study exists; it is explicitly the first of its kind.
  • A

    All-subject Exams

    • Only writing/grammar outcomes were assessed; no other core school subjects were measured.
    • "Two measures of grammar and writing were chosen, consistent with the hypothesis of the research that grammar teaching could improve pupils' writing."
    • Relevant Quotes: 1) "Two measures of grammar and writing were chosen, consistent with the hypothesis of the research that grammar teaching could improve pupils' writing." (p. 129) 2) "Progress in English (PiE) test: the primary outcome measure ... Sentence Generation Test (SG): the secondary outcome measure." (p. 129) 3) "Children's writing was measured at text-level and sentence-level." (p. 150) Detailed Analysis: Criterion A requires that the study assess impact across all main subjects taught, using standardised exams, to detect any unintended effects on non-intervention subjects. This study measured only writing outcomes (a standardised narrative writing test and a bespoke sentence generation test); no other core subjects such as mathematics or science were assessed. Even though the standardised primary outcome satisfies criterion E, the study does not cover all main subjects and offers no specialised-vocational justification for a single-subject focus. Criterion A is not met because outcomes were confined to writing/grammar, with no assessment of other core subjects.
  • G

    Graduation Tracking

    • Pupils were tested only one to two weeks after the intervention, with no tracking through to graduation.
    • "Post-intervention: one to two weeks after the end of the intervention."
    • Relevant Quotes: 1) "Pupils were tested twice on the writing outcome measures: 1. Baseline: prior to the start of the intervention; 2. Post-intervention: one to two weeks after the end of the intervention." (p. 130) 2) "We planned for additional delayed tests but due to COVID-19 this was not possible, hence a deviation from the published protocol." (p. 130) Detailed Analysis: Criterion G requires follow-up tracking of participants through to graduation from the relevant educational stage, and it also depends on the Year Duration criterion (Y) being met. Here outcomes were collected only one to two weeks after the ten-week intervention, planned delayed tests were abandoned, and there is no tracking of these Year 2 pupils to the end of primary school or graduation. No subsequent follow-up publication tracking this cohort to graduation was located (the pre-published protocol and SAP describe no graduation follow-up). Since criterion Y is not met, criterion G cannot be met either. Criterion G is not met because there was no long-term follow-up to graduation, and the year-duration prerequisite (Y) is also unmet.
  • P

    Pre-Registered

    • A full evaluation protocol (Anders et al., 2019) and statistical analysis plan were pre-published before data collection, and the trial was prospectively registered on the SREE Registry (ID #1920.1v1).
    • "An evaluation protocol and subsequent statistical analysis plan were published in protocol documents prior to completion of the research (Anders et al., 2021; Anders et al., 2019)."
    • Relevant Quotes: 1) "An evaluation protocol and subsequent statistical analysis plan were published in protocol documents prior to completion of the research (Anders et al., 2021; Anders et al., 2019)." (p. 124) 2) "Randomisation within each batch was, as reported in the pre-published protocol, carried out within stratification blocks to reduce the risk of imbalance on important characteristics..." (p. 127) 3) "The procedures for the quantitative data analyses of the test measures were pre-specified in a statistical analysis plan (SAP) (Anders et al., 2021)." (p. 133) 4) "Anders, J., Wyse, D., Aarts, B., Dockrell, J., Torgerson, C., Manyukhina, Y., Cushing, I., & Sing, S. (2019). Evaluation protocol: Grammar and Writing research project." (p. 153) Detailed Analysis: Criterion P requires that the study protocol, including hypotheses, methods and planned analyses, be pre-registered/published before data collection began. The paper cites a published evaluation protocol (Anders et al., 2019) and a statistical analysis plan (Anders et al., 2021), both hosted on UCL's public repository, and explicitly refers to the "pre-published protocol" and "pre-registered statistical analysis plan". The 2019 evaluation protocol was fetched and confirmed during verification (UCL Discovery eprint 10087427, "Grammar and Writing Evaluation Protocol"). It is written prospectively (the intervention "to be tested", recruitment/randomisation described as planned future activity), sets out the hypotheses, the school-level cluster RCT design, outcome measures and planned analyses, and includes an "Ethics and registration" section stating: "The trial is registered on the Registry of Efficacy and Effectiveness Studies with Registry ID: #1920.1v1". Its timeline places project start in January 2019 and Englicious training in December 2019/January 2020, and the paper dates the first randomisation batch to November 2019, so the protocol and registration precede data collection. The authors note one modelling deviation made on reviewer/ editorial guidance, but the pre-specified plan still exists and the reported estimates are described as highly similar to it. Criterion P is met because a full evaluation protocol and analysis plan were pre-published and the trial was prospectively registered (SREE Registry ID #1920.1v1) before data collection, as verified against the protocol document.