The impact of a story-based intervention on language, literacy, and cognitive development in South African pre-schoolers: randomised controlled trials for two language groups

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation occurred at the ECD-centre (site) level, which is stronger than class-level randomisation and satisfies criterion C.
    • "Early Childhood Development (ECD) centres were randomly assigned either to the treatment (Little Stars) condition or to a business-as-usual (existing curriculum) wait control condition."
    • Relevant Quotes: 1) "We conducted two separate RCTs: one comprising isiXhosa-speaking children and their educators, the other comprising Afrikaans-speaking children and their educators." 2) "Early Childhood Development (ECD) centres were randomly assigned either to the treatment (Little Stars) condition or to a business-as-usual (existing curriculum) wait control condition." 3) "One ECD centre in each language group had two classrooms and these classrooms were assigned to the same group." Detailed Analysis: Criterion C requires an RCT where allocation is at least class level (or stronger) to reduce contamination between treatment and control participants taught in the same classroom. The paper states that ECD centres (the implementing sites) were randomly assigned to treatment vs wait-list control, which is a stronger unit than class-level randomisation. The paper also clarifies that where an ECD centre contained two classrooms, both classrooms were assigned to the same condition, further reducing within-centre contamination risk. Final sentence: Criterion C is met because randomisation occurred at the ECD-centre level (stronger than class-level).
  • E

    Exam-based Assessment

    • The study uses ELOM, described as a standardised tool with established validity and reliability, meeting criterion E.
    • "The Early Learning Outcomes Measure (ELOM) is a standardised tool suitable for measuring the effects of early learning programmes and children’s readiness to learn in children aged 50–69 months (Snelling et al., 2019)."
    • Relevant Quotes: 1) "The Early Learning Outcomes Measure (ELOM) is a standardised tool suitable for measuring the effects of early learning programmes and children’s readiness to learn in children aged 50–69 months (Snelling et al., 2019)." 2) "Content and construct validity, reliability and cross-cultural fairness have been established (Anderson et al., 2021; Dawes et al., 2018)." 3) "Few standardised instruments for the assessment of general ability, language and literacy are available in South Africa, particularly in African languages." Detailed Analysis: Criterion E requires that outcomes are measured using a standardised, externally validated assessment rather than only researcher-created measures. The paper explicitly identifies ELOM as a "standardised tool" and states that validity, reliability, and cross-cultural fairness have been established. The paper also notes that some other measures were selected or developed due to limited availability of standardised instruments in the relevant languages, but the presence of ELOM as a key standardised assessment supports meeting this criterion. Final sentence: Criterion E is met because the study uses the standardised and validated ELOM as an outcome measure.
  • T

    Term Duration

    • Outcomes were measured at Time 2 after 26 weeks, which exceeds a typical academic term duration.
    • "Since ethics approval required the control group to access the intervention in the same calendar year as the intervention group, the evaluation was completed at 26 weeks in August 2022 (Time 2), rather than at 36 weeks on completion of the full programme, as originally planned."
    • Relevant Quotes: 1) "In February 2022 (Time 1) trained and accredited assessors conducted classroom observations and assessments of the children." 2) "The post-intervention assessment took place after 26 weeks." 3) "Since ethics approval required the control group to access the intervention in the same calendar year as the intervention group, the evaluation was completed at 26 weeks in August 2022 (Time 2), rather than at 36 weeks on completion of the full programme, as originally planned." Detailed Analysis: Criterion T requires outcomes to be measured at least one academic term after the intervention begins (typically ~3 to 4 months). The study reports a baseline at Time 1 (February 2022) and a follow-up at Time 2 after 26 weeks (August 2022). An interval of 26 weeks is substantially longer than one academic term, so the minimum duration requirement is satisfied, even though the full intended 36-week programme was not evaluated at completion. Final sentence: Criterion T is met because outcome measurement occurred after 26 weeks, exceeding a term-long duration.
  • D

    Documented Control Group

    • The paper documents the wait-list business-as-usual control group and reports group sizes and baseline characteristics, meeting D.
    • "We compared performance of an experimental group that received the classroom intervention to that of a (wait list) control group that received normal classroom teaching."
    • Relevant Quotes: 1) "Early Childhood Development (ECD) centres were randomly assigned either to the treatment (Little Stars) condition or to a business-as-usual (existing curriculum) wait control condition." 2) "We compared performance of an experimental group that received the classroom intervention to that of a (wait list) control group that received normal classroom teaching." 3) "The final sample for data analysis comprised: 82 (36 girls) isiXhosa-speaking and 118 (54 girls) Afrikaans-speaking participants across 20 isiXhosa (10 intervention, 10 control) and 24 Afrikaans ECD centres (12 intervention, 12 control)." 4) "Participant characteristics for the final sample by group are reported in Table 1." Detailed Analysis: Criterion D requires a clearly documented control group, including what the control condition received and sufficient information to assess baseline comparability. The paper clearly defines the control as a wait-list, business-as-usual condition receiving "normal classroom teaching" and provides sample sizes by arm and language group. It also points to baseline characteristics and outcomes tables, indicating the control group is documented and comparable information is provided. Final sentence: Criterion D is met because the control condition and its sample characteristics are clearly described and reported.

• Level 2 Criteria

  • S

    School-level RCT

    • ECD centres (sites) were randomly assigned to conditions, meeting the school/site-level RCT requirement.
    • "Early Childhood Development (ECD) centres were randomly assigned either to the treatment (Little Stars) condition or to a business-as-usual (existing curriculum) wait control condition."
    • Relevant Quotes: 1) "Early Childhood Development (ECD) centres were randomly assigned either to the treatment (Little Stars) condition or to a business-as-usual (existing curriculum) wait control condition." 2) "The final sample for data analysis comprised: 82 (36 girls) isiXhosa-speaking and 118 (54 girls) Afrikaans-speaking participants across 20 isiXhosa (10 intervention, 10 control) and 24 Afrikaans ECD centres (12 intervention, 12 control)." Detailed Analysis: Criterion S requires randomisation at the school level (or the relevant institutional unit implementing the programme). Here, ECD centres are the implementing sites and are explicitly stated to be randomly assigned to treatment vs control, with numbers of centres per arm reported. Final sentence: Criterion S is met because randomisation occurred at the ECD-centre (site) level.
  • I

    Independent Conduct

    • The paper does not clearly document evaluation independence from the intervention organisation/designers, and a competing interest is declared, so criterion I is not met.
    • "Shelley O’Carroll reports financial support was provided by Wordworks."
    • Relevant Quotes: 1) "Training for the teachers before and during the intervention was provided." 2) "We provided professional development to support reliable implementation of the intervention and collected independent assessments of the quality of implementation." 3) "Shelley O’Carroll reports financial support was provided by Wordworks." Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the intervention designers/providers, with clear documentation of role separation for implementation, measurement, and analysis. The paper describes provision of training and professional development as part of the intervention and mentions "independent assessments" of implementation quality, but it does not clearly state that outcome data collection and analysis were led by an external evaluation team independent from the intervention organisation/designers. The declaration of competing interests indicates a financial relationship (support from Wordworks) for an author connected to the programme context. Without explicit documentation of independent evaluation governance, this criterion is not satisfied. Final sentence: Criterion I is not met because independent conduct of the evaluation is not clearly documented.
  • Y

    Year Duration

    • Outcomes were assessed after 26 weeks, which is below 75% of the paper’s stated 36-week full school-year programme, so Y is not met.
    • "The programme was designed to last for the full school year (36 weeks)."
    • Relevant Quotes: 1) "The programme was designed to last for the full school year (36 weeks)." 2) "Since ethics approval required the control group to access the intervention in the same calendar year as the intervention group, the evaluation was completed at 26 weeks in August 2022 (Time 2), rather than at 36 weeks on completion of the full programme, as originally planned." Detailed Analysis: Criterion Y requires outcome measurement at least 75% of an academic year after the intervention begins. The paper defines the intended full school-year programme length as 36 weeks. Seventy-five percent of 36 weeks is 27 weeks. The evaluation occurred at 26 weeks, and the paper does not provide an alternative academic-year definition that would make 26 weeks meet the 75% threshold. Final sentence: Criterion Y is not met because follow-up occurred at 26 weeks, which is less than 75% of the stated 36-week year.
  • B

    Balanced Control Group

    • Although the intervention adds resources (materials and teacher training/support), these are integral to the package being tested against business-as-usual, so criterion B is met.
    • "Following this, they participated in two full-day training workshops and four further workshops (2.5 h each, held monthly between April and August 2022)."
    • Relevant Quotes: 1) "To address resourcing concerns and the sustainability of this project, teachers received professional development and support before and during the intervention to enhance their knowledge, and hard copies of the opensource resources needed to deliver the programme." 2) "Following this, they participated in two full-day training workshops and four further workshops (2.5 h each, held monthly between April and August 2022)." 3) "The programme comprises five activities per week with one daily teaching activity (in a two-week cycle/routine), which requires only 15–30 min per day and could be integrated into an existing daily teaching programme." 4) "The control group received the same training after the conclusion of the trial (from August to December 2022)." Detailed Analysis: Criterion B checks whether additional resources (time, materials, budget, staff inputs) are balanced across treatment and control, unless those extra resources are explicitly the treatment variable or are integral components of the intervention package. Extra resources are clearly present for the intervention arm: (a) hard-copy programme materials and (b) substantial teacher professional development (two full-day workshops plus monthly sessions). However, the paper defines the intervention as a programme package intended to be delivered by teachers using those specific resources and training, contrasted with a business-as- usual curriculum in a wait-list control. Under the criterion B decision tree, this fits the exception where the additional resources are integral to what is being tested (the Little Stars package versus business-as-usual), rather than a separable, unintended confound. Final sentence: Criterion B is met because the extra resources are integral to the intervention package being evaluated.

• Level 3 Criteria

  • R

    Reproduced

    • No independent, peer-reviewed replication by a different research team was identified, so criterion R is not met.
    • Relevant Quotes: 1) "We report two randomised controlled trials (RCTs) of the Little Stars story-based programme for literacy development in South African preschools located in the Western Cape, with separate trials for two different language groups." Detailed Analysis: Criterion R requires independent replication (a different research team, in a different context, peer-reviewed publication). The paper itself reports two trials (two language groups), but these are part of the same study team and publication and do not constitute an independent replication. Additional searching for later peer-reviewed papers describing an independent replication of this specific "Little Stars" programme (as evaluated here) did not identify a clear replication study by another team. Final sentence: Criterion R is not met because an independent peer-reviewed replication was not found.
  • A

    All-subject Exams

    • The study assesses broad learning domains (including emergent literacy/language and emergent numeracy) using the standardised ELOM, satisfying criterion A for broad coverage in this context.
    • "It assesses five domains: gross motor development, fine motor coordination and visual integration, emergent numeracy and mathematics, cognition and executive functioning, and emergent literacy and language (ELL)."
    • Relevant Quotes: 1) "The Early Learning Outcomes Measure (ELOM) is a standardised tool suitable for measuring the effects of early learning programmes and children’s readiness to learn in children aged 50–69 months (Snelling et al., 2019)." 2) "It assesses five domains: gross motor development, fine motor coordination and visual integration, emergent numeracy and mathematics, cognition and executive functioning, and emergent literacy and language (ELL)." Detailed Analysis: Criterion A requires assessment coverage broad enough to detect potential trade-offs (i.e., effects not only in the targeted area). In this pre-school context, ELOM provides a standardised, multi-domain assessment spanning emergent literacy/language and emergent numeracy/mathematics, as well as broader developmental domains. This constitutes broad "all-domain" coverage for the relevant educational stage, reducing the risk that gains in one area mask harms in another. Final sentence: Criterion A is met because the study uses the standardised ELOM to assess multiple core learning domains.
  • G

    Graduation Tracking

    • The study does not track participants to graduation, and because criterion Y is not met, criterion G is not met as well.
    • "Future studies should include follow-up assessments to determine longer-term impacts."
    • Relevant Quotes: 1) "Future studies should include follow-up assessments to determine longer-term impacts." 2) "Since ethics approval required the control group to access the intervention in the same calendar year as the intervention group, the evaluation was completed at 26 weeks in August 2022 (Time 2), rather than at 36 weeks on completion of the full programme, as originally planned." Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. The paper reports outcomes through Time 2 at 26 weeks and explicitly indicates that longer-term follow-up is for future studies, which implies no graduation tracking is reported in this paper. Additionally, per the ERCT rule: if criterion Y (Year Duration) is not met, then criterion G is not met. Since Y is not met here, G must be not met. Searches for subsequent follow-up papers by the same authors reporting graduation tracking for this cohort did not identify a peer-reviewed publication providing such tracking. Final sentence: Criterion G is not met because graduation tracking is not reported and Y is not met.
  • P

    Pre-Registered

    • The paper links to an OSF pre-registration but the registration date (and its precedence to data collection) is not documented in the paper, so criterion P is not met.
    • "The pre-registration of the study design and aims, and also the dataset and analytic code, can be found at https://osf.io/vhmj4."
    • Relevant Quotes: 1) "The pre-registration of the study design and aims, and also the dataset and analytic code, can be found at https://osf.io/vhmj4." 2) "In February 2022 (Time 1) trained and accredited assessors conducted classroom observations and assessments of the children." 3) "Different to our pre-registration, different vocabulary assessments were administered at pre- and post-test, as detailed below." Detailed Analysis: Criterion P requires that the full protocol be pre-registered before the study begins, with evidence of the registration date being before data collection. The paper provides an OSF link and states that the study was pre-registered, but it does not include (in the paper text) the OSF registration date or a statement explicitly confirming the registration date preceded Time 1 data collection in February 2022. The paper also notes at least one deviation from the pre- registration. Deviations are not automatically disqualifying, but they increase the importance of clearly documented registration timing and contents in the report itself. Final sentence: Criterion P is not met because the paper does not document a registration date proving pre-registration occurred before data collection.