Efficacy trial of Talking Time©: An oral language intervention for early years

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The trial randomised whole early years settings (nurseries/schools) to condition, a unit at least as strong as the class level, so the class-level RCT criterion is satisfied.
    • "The trial was a two-arm, setting-level RCT, with early years settings as the units of randomisation."
    • Relevant Quotes: 1) "The trial was a two-arm, setting-level RCT, with early years settings as the units of randomisation. Randomisation took place during the week of 04 November 2024... A total of 123 settings were randomised: 62 to the intervention group; and 61 to the control group." (Methods, "Randomisation") 2) "The evaluation employs a two-arm, cluster RCT design, with early years settings serving as the unit of randomisation." (Methods, "Trial design") Detailed Analysis: The paper is explicit that randomisation occurred at the level of the entire early years setting (nursery, school-based nursery, or PVI), not at the level of individual students or single classes within a setting. Under the ERCT Standard, class-level RCT is the base requirement, and a stronger school/setting-level RCT automatically satisfies it, since setting-level randomisation eliminates the risk of within-classroom contamination between treatment and control children. This was independently re-verified against the source PDF and both quotes are verbatim. Final sentence: Criterion C is met because the whole-setting unit of randomisation exceeds the class-level requirement.
  • E

    Exam-based Assessment

    • Outcomes were measured using the Renfrew Expressive Vocabulary Test, the Renfrew Action Picture Test, and the GAPS test, all commercially published, widely used standardised assessments with UK age norms.
    • "The Renfrew assessments are widely used in applied research in the UK."
    • Relevant Quotes: 1) "Children's language outcomes were measured at baseline and at follow-up using–Renfrew tests (expressive vocabulary and action picture), and the Grammar and Phonology Screening (GAPS) test." (Executive summary) 2) "The Renfrew assessments are widely used in applied research in the UK... The Renfrew scores are available in percentile form against an age-equivalent sample norm from the UK." (Methods, "Primary outcome") 3) "The GAPS test is validated for children aged three years and four months to six years and six months, and reliably identifies language impairment (van der Lely et al., 2011)... The instrument is publicly available online." (Methods, "Secondary outcomes") Detailed Analysis: The Renfrew Expressive Vocabulary Test (REV, 5th edition, 2023) and Renfrew Action Picture Test (RAPT, 5th edition, 2019) are commercially published, standardised instruments with UK norms, not custom-built for this study. The GAPS test is similarly a validated, standardised, and publicly documented instrument with reported internal consistency and diagnostic validity. None of the three instruments were created by the research team to specifically match the intervention content. All three quotes were re-checked against the source PDF and are verbatim. Final sentence: Criterion E is met because all primary and secondary outcome measures are established, standardised, non-custom assessments.
  • T

    Term Duration

    • Because the stronger Year Duration criterion (Y) is met (outcomes measured roughly eight months after intervention start, close to a full academic year), the weaker Term Duration criterion is automatically considered met.
    • "...intervention delivery from November 2024 to June 2025, and follow-up assessments completed by July 2025."
    • Relevant Quotes: 1) "The trial ran from autumn 2024, with baseline assessments conducted between September and November 2024, intervention delivery from November 2024 to June 2025, and follow-up assessments completed by July 2025." (Executive summary) Detailed Analysis: The interval between intervention start (11 November 2024) and follow-up outcome measurement (completed by July 2025) is approximately eight months, far exceeding the minimum one-term (3-4 month) requirement. Per the ERCT Standard, satisfying the stronger Year Duration criterion automatically satisfies this weaker criterion. Note: the top-level "quote" field was corrected from the agent's original version, which incorrectly capitalised "Intervention" as if it were a stand-alone sentence; a leading ellipsis has been added to preserve verbatim casing. Final sentence: Criterion T is met because the measurement interval greatly exceeds one academic term and the stronger Year Duration criterion is also met.
  • D

    Documented Control Group

    • The control group's characteristics, size, demographics, and lack of programme exposure are documented in detail, including baseline comparison tables against the intervention group.
    • "Setting-level and child-level characteristics are very similar across the intervention and the control groups, indicating that randomisation was successful."
    • Relevant Quotes: 1) "Settings allocated to the control arm do not receive the programme during the trial period and are expected to continue with business as usual practice." (Methods, "Trial design") 2) "Setting-level and child-level characteristics are very similar across the intervention and the control groups, indicating that randomisation was successful." (Impact evaluation results, "Children and setting characteristics") 3) Table 12 reports, for both intervention and control groups: proportion of PVIs, IDACI deprivation score, EYPP eligibility, EAL status, gender, age, and baseline scores on all four assessments, with sample sizes and standard deviations for each group. Detailed Analysis: The paper provides an extensive, quantified comparison of control-group characteristics (demographics, baseline attainment, setting type, deprivation index) against the intervention group, alongside qualitative description of control settings' "business as usual" practice from the IPE. Both direct quotes were re-checked against the source PDF and are verbatim. Final sentence: Criterion D is met because control group demographics, baseline performance, and conditions are thoroughly documented and compared to the intervention group.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the level of the early years setting (nursery school, school-based nursery, or PVI), which is the educational institution implementing the intervention, satisfying the school-level RCT requirement.
    • "A total of 123 settings were randomised: 62 to the intervention group; and 61 to the control group."
    • Relevant Quotes: 1) "The trial was a two-arm, setting-level RCT, with early years settings as the units of randomisation... A total of 123 settings were randomised: 62 to the intervention group; and 61 to the control group." (Methods, "Randomisation") 2) "A stratified randomisation design was used, with 12 strata defined by six geographic regions... crossed with PVI status." (Methods, "Randomisation") Detailed Analysis: Under the ERCT Standard, 'school' includes any educational institution or unit implementing the intervention, explicitly including preschool centres. Here, 123 whole nursery settings (maintained nurseries, school-based nurseries, and PVIs) were randomised as intact units, stratified by region and setting type, which meets the requirement of randomising at the level of the implementing institution rather than at the class or student level. Cross-checking the companion evaluation protocol document confirms the same design: "Randomisation will take place at setting level" (Protocol, "Randomisation"), corroborating that setting-level randomisation was the intended and executed design, not a post-hoc description. Final sentence: Criterion S is met because randomisation was conducted at the level of the whole early years setting.
  • I

    Independent Conduct

    • The evaluation (data collection, analysis, and reporting) was conducted by an independent research team (NIESR), separate from the delivery team (University of Oxford and UCL) that designed and developed the intervention. An advisory-only role held by the original programme developer does not undermine this independence.
    • "The project was independently evaluated by a team from the National Institute of Economic and Social Research including Edoardo Masset, Anisa Butt, and Sherelle Davids."
    • Relevant Quotes: 1) "The project was independently evaluated by a team from the National Institute of Economic and Social Research including Edoardo Masset, Anisa Butt, and Sherelle Davids. Edoardo Masset was the project lead." ("About the evaluator") 2) "Talking Time© was initially developed by the Institute of Education at UCL... The University of Oxford subsequently created a more comprehensive manual... and developed the structured professional development programme." (Intervention, "Who (provider)") 3) "Delivery team (University of Oxford and UCL)... Sandra Mathers, Associate Professor, University of Oxford. Principal investigator for the delivery team." ("Project team") 4) "Baseline assessments took place primarily before randomisation, ensuring that assessors were blind to treatment allocation... At follow-up assessors remained blinded to treatment assignment and setting staff were instructed not to disclose the assignment status of the setting to the assessors." (Methods, "Outcome measures") 5) "The project team supported the study with comments in an advisory role all along the project and included Claudine Bowyer-Crane (University of Sheffield), Julie Dockrell (University College London), Ignacia Valenzuela (the Education Endowment Foundation [EEF]), and Aoife Duff (EEF)." (Acknowledgements) Detailed Analysis: The programme was designed and delivered by the University of Oxford/UCL delivery team, while data collection, analysis, and report authorship were conducted by a separate organisation, NIESR, with its own principal investigator. Assessors were blinded to allocation at both baseline and follow-up, further reducing the risk of biased measurement. On closer scrutiny, quote 5 shows that Julie Dockrell (the intervention's original developer at UCL) held an "advisory role" on the project team throughout the study, alongside a Sheffield-based NIESR Fellow and two EEF staff. This is a genuine nuance worth flagging, but per the ERCT Standard's own worked examples (e.g. a contracted author who does not participate in data collection, analysis, or conclusions can still satisfy Criterion I), an advisory/commentary role that is explicitly separate from data collection, analysis, and report-writing does not undermine independent conduct. The report consistently frames NIESR alone as responsible for holding and analysing the data and writing the report, and Sandra Mathers' role (Oxford, delivery team PI) is confined to intervention design, delivery oversight, and recruitment/consent processes, not analysis. Final sentence: Criterion I is met because an independent evaluation organisation (NIESR), separate from the intervention's developers, conducted the data collection and analysis with blinded assessors, and the one identified advisory link to the original developer did not extend to data collection, analysis, or conclusions.
  • Y

    Year Duration

    • Outcomes were measured roughly eight months after intervention start (November 2024 to June/July 2025), which is within 75% of a full academic year and therefore satisfies the Year Duration criterion's minor-shortfall allowance.
    • "...intervention delivery from November 2024 to June 2025, and follow-up assessments completed by July 2025."
    • Relevant Quotes: 1) "Randomisation took place during the week of 04 November 2024, one week prior to the intervention start (11 November 2024)." (Methods, "Randomisation") 2) "The trial ran from autumn 2024... intervention delivery from November 2024 to June 2025, and follow-up assessments completed by July 2025." (Executive summary) 3) "Programme delivery occurred from late 2024 to summer 2025 on a three-week rolling basis to support practical implementation." (Introduction, "When and how much (dosage)") 4) "Follow-up assessments were conducted between June and July 2025." (Impact evaluation results, "Participant flow") Detailed Analysis: The intervention began on 11 November 2024, and follow-up assessments (the primary outcome measurement) were conducted between June and July 2025, an interval of approximately seven to eight months. A full UK academic year runs roughly September to July (about 9-10 months per the ERCT Standard's own definition), so this corresponds to roughly 75-85% of a full academic year. This sits at or above the 75% threshold the ERCT Standard treats as an acceptable "technical reasons" shortfall. Re-checking the exact assessment window (quote 4: June-July 2025, not just July) narrows the estimate slightly compared with the original ranking but the interval still clears the 75% threshold under the Standard's own ~9-10 month definition of an academic year. The shortfall (starting in November rather than September, for evaluation logistics) is minor and does not indicate the study reverted to only a term-length design. Final sentence: Criterion Y is met because the interval from intervention start to outcome measurement covers approximately 75-85% of a full academic year, at or above the standard's allowance for minor shortfalls.
  • B

    Balanced Control Group

    • The additional professional development and small-group session time are the explicit treatment variable being tested against a "business as usual" control, which the ERCT Standard treats as a valid design rather than an imbalance.
    • "Settings allocated to the intervention arm received the full Talking Time© programme, delivered with support and oversight... Settings allocated to the control arm do not receive the programme during the trial period and are expected to continue with business as usual practice."
    • Relevant Quotes: 1) "This study is an efficacy trial, designed to test how the Talking Time© intervention works under ideal and controlled conditions." (Methods, "Trial design") 2) "Settings allocated to the intervention arm received the full Talking Time© programme, delivered with support and oversight to ensure adherence to its core components. Settings allocated to the control arm do not receive the programme during the trial period and are expected to continue with business as usual practice." (Methods, "Trial design") 3) "Control settings receive a £1,000 incentive payment at the end of the trial... All participating settings, regardless of allocation, also received a £400 thank you payment for supporting the research activities." (Methods, "Trial design") 4) "Children participate in two 15-minute structured oral language sessions per week, for 20 weeks, in small groups of up to five children of mixed language ability." (Intervention, "Programme for children") 5) "Sessions were delivered by specifically trained early years setting staff (usually two leads per setting in the current study), supported by a structural professional development programme that included training, mentoring, and ongoing reflective practice." (Executive summary) Detailed Analysis: Applying the criterion B decision tree: extra resources are present (small-group sessions and a structured professional development programme for staff). However, these additional resources are not a supplementary add-on to an otherwise unchanged core practice; they are themselves the explicit treatment variable this efficacy trial is designed to test ("designed to test how the Talking Time© intervention works under ideal and controlled conditions"). The control group continues ordinary "business as usual" practice, as intended by the design, and both arms receive identical financial thank-you payments so that money is not a confound. This falls squarely under the RESOURCES_ARE_TREATMENT branch of the ERCT Standard's Criterion B decision tree: since the whole point of the trial is to test whether the integrated Talking Time© package (sessions plus professional development) improves outcomes, the control arm's "business as usual" status is the correct comparator by design, not a violation. Correction note: the original ranking's quote 4 spliced together wording from two different locations in the paper (the Intervention section and the Executive summary) under a single incorrect "(Executive summary)" citation; this has been split into two separately and correctly sourced quotes (4 and 5) above. The original ranking's analysis also referenced a "STRIPES2/para-instructor example" that does not appear anywhere in the ERCT Standard; that fabricated reference has been removed and replaced with a direct citation of the Standard's actual Criterion B decision tree. Final sentence: Criterion B is met because the additional professional development and session time are the intervention being tested, and the control group's business-as-usual condition is the appropriate comparator by design.

• Level 3 Criteria

  • R

    Reproduced

    • No evidence is presented that this specific efficacy trial, or an equivalent independent trial in a comparable design, has been replicated by a separate research team in a peer-reviewed outlet.
    • "A subsequent randomised controlled trial (RCT) involving 40 settings, funded by the Nuffield Foundation in 2019/2020, was halted due to the COVID-19 pandemic. A retrial in 2021/2022 with 36 settings (Dockrell et al., 2023)..."
    • Relevant Quotes: 1) "An initial evaluation of the programme (Dockrell et al., 2010), conducted in three London nurseries in a highly deprived borough, employed a quasi-experimental design..." (Introduction, "Background") 2) "A subsequent randomised controlled trial (RCT) involving 40 settings, funded by the Nuffield Foundation in 2019/2020, was halted due to the COVID-19 pandemic. A retrial in 2021/2022 with 36 settings (Dockrell et al., 2023) assessed impacts on expressive vocabulary, grammar, comprehension, oral narrative skills..." (Introduction, "Background") 3) "Julie Dockrell, Professor of Psychology and Special Needs at UCL, Institute of Education... Developed the original Talking Time© programme and led the first three evaluations of the programme." ("Project team", Delivery team) Detailed Analysis: Prior evaluations of Talking Time© exist (Dockrell et al., 2010; the halted 2019/2020 Nuffield trial; the 2021/2022 retrial, Dockrell et al., 2023), but the programme's original developer, Julie Dockrell, "led the first three evaluations of the programme," so these were not independent replications by a separate research team; they were developer-led studies of earlier versions of the intervention. This is independently corroborated by the companion evaluation protocol document, which states Julie Dockrell "developed the original Talking Time© programme and led its first evaluation" and "was principal investigator on the recent randomised controlled trial of Talking Time©, funded by the Nuffield Foundation" (Protocol, "Personnel"), confirming she led both the original quasi-experimental study and the Nuffield-funded RCT work. The current NIESR-led efficacy trial is itself the first fully independent evaluation and, being newly published (July 2026), has not yet been reproduced by any other, unconnected research team in a peer-reviewed outlet. No internet-searchable independent replication of this specific efficacy trial or its design was identified (note: general web search was unavailable during this verification pass because the session's search budget was exhausted; direct page fetches of the EEF website were also blocked by the site returning HTTP 403 to automated requests, so this remains an internet-informed but not fully internet-confirmed absence). Final sentence: Criterion R is not met because the only prior trials of Talking Time© were led or co-led by the programme's original developer rather than by a fully independent team, and this current trial has not itself been independently replicated.
  • A

    All-subject Exams

    • The study measured only oral language outcomes (vocabulary, grammar, information, sentence repetition) and did not assess other developmental or curriculum domains such as early numeracy.
    • "Children's language outcomes were measured at baseline and at follow-up using–Renfrew tests (expressive vocabulary and action picture), and the Grammar and Phonology Screening (GAPS) test."
    • Relevant Quotes: 1) "Children's language outcomes were measured at baseline and at follow-up using–Renfrew tests (expressive vocabulary and action picture), and the Grammar and Phonology Screening (GAPS) test." (Executive summary) 2) Table 3 in the Methods section ("Trial design") lists the primary outcome as oral language skills (a composite of expressive vocabulary and of information and grammar) and the four secondary outcomes as sentence repetition, expressive vocabulary, information, and grammar -- all facets of oral language, with no other curriculum subject represented. 3) "Oral language is a multidimensional construct encompassing vocabulary, grammar, narrative, and broader expressive language competencies." (Methods, "Primary outcome") Detailed Analysis: Criterion E is met (standardised assessments were used), so this criterion turns on whether all main developmental/curriculum domains for the target age group were assessed. Every primary and secondary outcome in this trial (REV, RAPT information, RAPT grammar, GAPS sentence repetition) measures a facet of oral language only. No assessment of early numeracy, or any other Early Years Foundation Stage learning area, is reported, even though such domains are commonly assessed alongside language in comparable early years trials (e.g., EGRA and EGMA together in similarly aged cohorts elsewhere). No rationale is given for excluding other core early learning domains as a deliberate, justified specialisation (the "vocational education" style exception in the Standard does not naturally apply to a universal nursery language programme). Note: quote 2 has been reworded as a description of Table 3's content rather than presented as a direct verbatim quote, since it summarises a table rather than quoting continuous prose. Final sentence: Criterion A is not met because only oral language outcomes were assessed, with no measurement of other core early learning domains and no justification offered for this narrow scope.
  • G

    Graduation Tracking

    • Follow-up tracking stopped a few months after the intervention ended and focused on practitioner sustainability, not on children's outcomes through to graduation from their educational stage.
    • "A series of interviews with practitioners were undertaken in 2026, several months after the end of the trial, to understand: whether and how Talking Time© continues to be delivered..."
    • Relevant Quotes: 1) "Follow-up assessments were conducted between June and July 2025." (Impact evaluation results, "Participant flow") 2) "A series of interviews with practitioners were undertaken in 2026, several months after the end of the trial, to understand: whether and how Talking Time© continues to be delivered; the extent to which core principles are maintained or adapted; the factors influencing how the programme is embedded over time; and any perceived lasting impact on children's oral language development; and staff practices." (Methods, "Follow-up interviews") 3) "Semi-structured interviews were conducted with six settings between February and April 2026... all of which reported continuing Talking Time© activities." (Methods, "Follow-up interviews") Detailed Analysis: Child outcome data collection ended at the follow-up assessment in June/July 2025, immediately after the intervention period. The only later follow-up activity, conducted in early-to-mid 2026, consisted of qualitative practitioner interviews about programme sustainability and perceived impact, not further child language assessment, and did not track the same cohort of children through to the end of their nursery stage, Reception, or any later graduation point. No child-level graduation tracking or later child outcome follow-up paper is referenced. Given how recently this trial concluded (data collection ended in 2025; report published July 2026), it is not plausible that a graduation-tracking follow-up study of this specific cohort could yet exist, and none was found. Final sentence: Criterion G is not met because there is no tracking of children's outcomes through to graduation; the only 2026 follow-up concerned practitioner sustainability rather than child language outcomes.
  • P

    Pre-Registered

    • The trial's registry number is reported and is confirmed by the companion protocol document to correspond to this exact trial, but no quoted date confirms pre-registration occurred before data collection began, so the timing requirement cannot be verified.
    • "The RCT was registered with the International Standard Randomized Controlled Trial Number (ISRCTN) UK Clinical Study Registry with number ISRCTN84530173."
    • Relevant Quotes: 1) "The RCT was registered with the International Standard Randomized Controlled Trial Number (ISRCTN) UK Clinical Study Registry with number ISRCTN84530173." (Introduction, "Ethics and trial registration") 2) "Protocol and statistical plan of the study can be found at the EEF website at the following links..." (footnote 2, Introduction) 3) "Pre-intervention data collection and baseline assessments" is listed in the project timeline as occurring "August 2024 – October 2024" (Methods, "Timeline", Table 9), before the "11 November 2024" randomisation date, but no corresponding date is given in the text for when ISRCTN registration itself took place. 4) "The trial will be registered at the ISRCTN registry; the registration number will be added to the protocol once this becomes available." (Companion Evaluation Protocol document, "Ethics and registration", version 1.0, dated July 2024) 5) "The protocol was registered with the ISRCTN registry with code ISRCTN84530173 (ISRCTN - ISRCTN84530173: Evaluation of Talking Time: efficacy trial)" (Companion Evaluation Protocol document, "Study rationale and background") Detailed Analysis: The paper confirms registration on a recognised public registry (ISRCTN) and references a published protocol and statistical analysis plan, which is positive evidence of transparency. To verify this further, the companion evaluation protocol document for this trial was retrieved and read in full. It confirms (quote 5) that ISRCTN84530173 specifically corresponds to "Evaluation of Talking Time: efficacy trial", ruling out a mismatched or generic registry code. However, the same protocol document's own "Ethics and registration" section, unchanged from its original July 2024 version, states the registration number was still pending ("will be added to the protocol once this becomes available") at a point in time after settings recruitment had already been underway for roughly six months (recruitment began January 2024, per the protocol timeline) and shortly before baseline data collection began (August 2024). This raises a genuine, unresolved question as to whether formal ISRCTN registration was completed before or after recruitment and/or baseline data collection started. An attempt was made to retrieve the exact registration date directly from the ISRCTN registry (isrctn.com/ISRCTN84530173) to resolve this, but the registry's page could not be retrieved in a readable form by the available web-fetching tool, and the session's web-search budget was exhausted before this could be pursued further. In the absence of a verifiable registration date, the criterion cannot be confirmed as met, consistent with the original ranking's conclusion, though the underlying uncertainty is now better documented and the registry code itself is confirmed genuine. Final sentence: Criterion P is not met because, although a registry ID confirmed to match this trial and a protocol are cited, no verifiable registration date could be obtained (from the paper or from the registry itself) to confirm that pre-registration occurred before data collection began.