News literacy education and civic engagement among primary school children: a mixed-methods evaluation

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Schools were the unit of randomisation, meeting (and exceeding) the class-level RCT requirement.
    • "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) using the excel macro ‘¼ RANDBETWEEN (1,2)’, such that there were 20 schools per condition."
    • Relevant Quotes: 1) "We used a cluster randomised controlled trial design to select multiple schools that were randomised to one condition or another, with only one group receiving the intervention." (p. 7) 2) "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) using the excel macro ‘¼ RANDBETWEEN (1,2)’, such that there were 20 schools per condition." (p. 4) Detailed Analysis: Criterion C requires an RCT with randomisation at the class level or a stronger unit (e.g., school), to reduce contamination between treatment and control. The paper explicitly states the study used a "cluster randomised controlled trial design" and that "schools were randomly allocated" to intervention and control conditions. Because randomisation occurs at the school level (a stronger unit than classroom-level randomisation), this satisfies Criterion C. Criterion C is met because schools (clusters) were randomly allocated to intervention and control conditions.
  • E

    Exam-based Assessment

    • Outcomes relied on adapted self-report items and a bespoke performance test rather than widely recognised standardised exams.
    • "At the outset of this study, there were no measures of news literacy that had been validated for the age group in question."
    • Relevant Quotes: 1) "At the outset of this study, there were no measures of news literacy that had been validated for the age group in question. Therefore, existing measures that were used with older children (e.g., Ashley et al., 2013; Maksl et al., 2017; Vraga et al., 2015) were adapted to use language that we thought would be relevant and accessible for children in our study." (p. 7) 2) "To supplement the self-report measures concerned with the evaluation of news stories (dimensions 3 and 5 above), a twelve-item performance test (see Figure 3 for an example item) was used to measure participants’ ability to recognise ‘fake news’." (p. 7) 3) "This test, hereinafter referred to as the ‘Detecting Fake News’ test, involved twelve online news items." (p. 7) Detailed Analysis: Criterion E requires outcome measurement using standardised, widely recognised exam-based assessments (e.g., national/state exams or other established standardised tests), rather than study-specific or researcher-adapted measures. The paper states there were no validated news-literacy measures for this age group and that the team adapted instruments used with older cohorts. It also describes a bespoke twelve-item "Detecting Fake News" performance test built for this evaluation. None of these are described as official standardised exams used broadly outside the study. Criterion E is not met because the study does not use standardised exam-based assessments.
  • T

    Term Duration

    • Outcomes were measured again at week 16 (three months after the intervention), which is roughly one academic term after baseline.
    • "By week 16, when the follow-up survey was administered, the final matched samples for T1 and Time 2 (T2) were 353 for the intervention group and 302 for the control group ..."
    • Relevant Quotes: 1) "Teachers then delivered a unit of work consisting of 15 one-hour lessons. This unit typically took between three and five weeks to complete, depending on how schools incorporated it into their timetables ..." (p. 5) 2) "There were three measurement time points: (i) immediately prior to the intervention, (ii) one month later immediately following the intervention, and (iii) three months after the intervention." (p. 7) 3) "By week 16, when the follow-up survey was administered, the final matched samples for T1 and Time 2 (T2) were 353 for the intervention group and 302 for the control group ..." (p. 6) Detailed Analysis: Criterion T requires that outcomes be measured at least one academic term after the intervention begins (typically ~3–4 months). The paper describes a follow-up at "week 16" and also describes the third time point as "three months after the intervention." While the intervention itself is described as taking "between three and five weeks to complete," the presence of a week-16 follow-up indicates outcome measurement at roughly a term-scale interval from the baseline period immediately prior to the intervention. Criterion T is met because the study includes a follow-up measurement at week 16, which is approximately a school term after baseline.
  • D

    Documented Control Group

    • The control group is described as curriculum-as-usual and baseline characteristics are reported by condition.
    • "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) ..."
    • Relevant Quotes: 1) "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) ..." (p. 4) 2) "In the NewsWise condition, 33.28% of participants were eligible for FSM, while in the Control condition it was 39.05%; both were well above the English national average of 22% from 2022." (p. 4) 3) "Most participants were aged 9 (n ¼ 664), 10 (n ¼ 1,045), or 11 (n ¼ 249) ..." (p. 4) Detailed Analysis: Criterion D requires that the control group be clearly documented, including what the control condition was and baseline characteristics enabling comparison. The paper explicitly defines the control condition as "curriculum as usual," provides the control group size, and reports baseline demographic and contextual indicators (e.g., age distribution and FSM eligibility by condition). This constitutes meaningful documentation of who was in the control group and what they received. Criterion D is met because the control group condition and baseline characteristics are clearly reported.

• Level 2 Criteria

  • S

    School-level RCT

    • Schools were randomised to condition, satisfying the school-level RCT criterion.
    • "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) ..."
    • Relevant Quotes: 1) "We used a cluster randomised controlled trial design to select multiple schools that were randomised to one condition or another, with only one group receiving the intervention." (p. 7) 2) "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) using the excel macro ‘¼ RANDBETWEEN (1,2)’, such that there were 20 schools per condition." (p. 4) Detailed Analysis: Criterion S requires randomisation at the school level (not just class or student level). The paper explicitly states the design is cluster-based and that "schools were randomly allocated" to intervention and control. This directly satisfies the school-level RCT requirement. Criterion S is met because schools were the unit of randomisation.
  • I

    Independent Conduct

    • The trial protocol explicitly distinguishes the Guardian Foundation as developer and universities as evaluators and states the trial was conducted by an independent academic team.
    • "This randomised cluster control trial will build on the promising NLT evaluation in the following ways: • It will be conducted by an independent academic team;"
    • Relevant Quotes (Paper): 1) "Funded by the Nuffield Foundation, the evaluation of NewsWise was conducted by the authors of this article in partnership with the Guardian Foundation, the National Literacy Trust, and the PSHE Association." (p. 4) Relevant Quotes (Trial Protocol): 2) "DEVELOPER (INSTITUTION) Guardian Foundation with the PSHE Association" (Trial protocol, p. 1) 3) "EVALUATOR (INSTITUTION) University of Birmingham (UoB), University of Liverpool (UoB), National Literacy Trust" (Trial protocol, p. 1) 4) "This randomised cluster control trial will build on the promising NLT evaluation in the following ways: • It will be conducted by an independent academic team;" (Trial protocol, p. 3) Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the intervention developer/design team, with clear documentation of that separation. The paper notes the evaluation was conducted "in partnership with" the Guardian Foundation and other organisations, which by itself could be read as a close collaboration. However, the publicly available trial protocol explicitly labels the Guardian Foundation (with PSHE Association) as the "DEVELOPER" and separately lists the "EVALUATOR" institutions as the universities and the National Literacy Trust, and it explicitly states the trial "will be conducted by an independent academic team." Given these explicit statements in the study protocol, the study provides documentary support that the evaluation team was independent of the programme developer. Criterion I is met because the study documentation explicitly states the evaluation was conducted by an independent academic team separate from the programme developer.
  • Y

    Year Duration

    • The longest follow-up described is week 16 (around four months), which is far short of 75% of an academic year.
    • "There were three measurement time points: (i) immediately prior to the intervention, (ii) one month later immediately following the intervention, and (iii) three months after the intervention."
    • Relevant Quotes: 1) "There were three measurement time points: (i) immediately prior to the intervention, (ii) one month later immediately following the intervention, and (iii) three months after the intervention." (p. 7) 2) "By week 16, when the follow-up survey was administered, the final matched samples for T1 and Time 2 (T2) were 353 for the intervention group and 302 for the control group ..." (p. 6) Detailed Analysis: Criterion Y requires outcomes to be measured at least 75% of an academic year after the intervention begins (typically ~9–10 months, allowing up to ~25% shorter). The paper reports measurement up to "week 16" and frames the follow-up as "three months after the intervention." Week 16 (about four months) is well below a year-scale tracking window, and the paper does not describe any later year-end or end-of-school-year outcome collection. Criterion Y is not met because outcomes are only tracked to week 16 rather than for most of an academic year.
  • B

    Balanced Control Group

    • The intervention provides additional structured learning and supports, and these added inputs are integral to the NewsWise treatment package being evaluated.
    • "The NewsWise programme consists of 15 lessons delivered over a 6-week period, as well as a live workshop from the NewsWise team and interactions with a journalist. In total, this involves 20 hours of learning, embedded within existing class time."
    • Relevant Quotes (Paper): 1) "Schools were randomly allocated to the experimental (NewsWise intervention; n ¼ 988) or control group (curriculum as usual; n ¼ 984) ..." (p. 4) 2) "All teachers took part in a training session delivered by the NewsWise team (one hour and fifteen minutes), followed by all classes receiving a two-hour workshop delivered by the same team. Teachers then delivered a unit of work consisting of 15 one-hour lessons." (p. 5) Relevant Quotes (ISRCTN record summary): 3) "The NewsWise programme consists of 15 lessons delivered over a 6-week period, as well as a live workshop from the NewsWise team and interactions with a journalist. In total, this involves 20 hours of learning, embedded within existing class time." (ISRCTN13350949 summary) Detailed Analysis: Criterion B assesses whether the control condition offers comparable time and resources, unless the additional inputs are integral to what is being tested. The paper documents that the intervention includes teacher training, a live workshop, and a structured 15-lesson unit. The control condition is described as "curriculum as usual," which does not mirror these specific inputs. External study documentation (ISRCTN summary) also describes the package as "20 hours of learning," reinforcing that the programme includes substantial structured inputs. Under the ERCT Criterion B decision logic, these additional resources are not accidental add-ons; they are core components of the NewsWise intervention package being evaluated against business-as-usual schooling. Therefore, the imbalance is part of the treatment definition rather than an uncontrolled confound. Criterion B is met because the additional time and supports are integral to the intervention package being tested.

• Level 3 Criteria

  • R

    Reproduced

    • No independent, peer-reviewed replication of this specific NewsWise RCT was identified.
    • Relevant Quotes: 1) "The NewsWise project run from March 2022 to March 2024, with the fieldwork taking place from September 2022 to June 2023." (p. 10) Detailed Analysis: Criterion R requires an independent replication (by a different research team in a different context) published in a peer-reviewed journal. The paper describes the original evaluation and its fieldwork period but does not claim that an independent replication has occurred. Additional internet searching for replication attempts (e.g., using the programme name, authors, and trial registry identifier) did not surface a distinct peer-reviewed replication study by an independent team that explicitly reproduces this RCT. Criterion R is not met because independent peer-reviewed replication evidence for this specific trial was not found.
  • A

    All-subject Exams

    • Because standardised exams were not used (E not met), the all-subject standardised exam requirement is not met.
    • Relevant Quotes: 1) "At the outset of this study, there were no measures of news literacy that had been validated for the age group in question. Therefore, existing measures that were used with older children (e.g., Ashley et al., 2013; Maksl et al., 2017; Vraga et al., 2015) were adapted ..." (p. 7) Detailed Analysis: Criterion A requires standardised, exam-based assessment across all main subjects and additionally depends on Criterion E being met. Since this study does not use standardised exam-based assessments (Criterion E is not met), it cannot satisfy Criterion A regardless of subject coverage. Criterion A is not met because Criterion E is not met.
  • G

    Graduation Tracking

    • The study does not track pupils through graduation, and because Y is not met, G cannot be met under the ERCT rules.
    • Relevant Quotes: 1) "There were three measurement time points: (i) immediately prior to the intervention, (ii) one month later immediately following the intervention, and (iii) three months after the intervention." (p. 7) 2) "By week 16, when the follow-up survey was administered, the final matched samples for T1 and Time 2 (T2) were 353 for the intervention group and 302 for the control group ..." (p. 6) Detailed Analysis: Criterion G requires tracking participants until graduation. The paper’s outcomes are measured only up to the three-month follow-up (week 16), and it contains no description of tracking the cohort through primary-school completion (or any other graduation endpoint). Additionally, the ERCT rule provided states that if Criterion Y is not met, then Criterion G is not met. Since the study does not meet the year-duration requirement, it cannot satisfy graduation tracking. Criterion G is not met because there is no evidence of tracking until graduation and the year-duration prerequisite is not met.
  • P

    Pre-Registered

    • The study is registered (ISRCTN13350949), but the registry shows registration occurred after the registry’s date of first enrolment.
    • "Registration date 14/09/2022"
    • Relevant Quotes (Paper): 1) "In accordance with the trial protocol,1 there were two random factors to be controlled for during the analysis using hierarchical linear modelling: School and Participant." (p. 7) Relevant Quotes (Trial Protocol): 2) "The evaluation is registered with the ISRCTN (International Standard Randomised Controlled Trial Number) number: ISRCTN13350949" (Trial protocol, p. 12) Relevant Quotes (ISRCTN registry record summary): 3) "Submission date 05/09/2022" (ISRCTN13350949 record summary) 4) "Registration date 14/09/2022" (ISRCTN13350949 record summary) 5) "Date of first enrolment 01/06/2022" (ISRCTN13350949 record summary) Detailed Analysis: Criterion P requires a publicly accessible pre-registered protocol with a verifiable registration reference and evidence that registration occurred before the study began (i.e., before enrolment / data collection starts). The paper references a trial protocol. The protocol explicitly states the evaluation is registered as ISRCTN13350949. The ISRCTN record summary provides a "Registration date 14/09/2022" and also lists "Date of first enrolment 01/06/2022." On the face of these dates, the registry indicates registration occurred after enrolment began. Criterion P is not met because the registry dates indicate registration occurred after the listed date of first enrolment.