Does computer-assisted learning improve learning outcomes? Evidence from a randomized experiment in migrant schools in Beijing

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized whole classes within schools, satisfying the class‑level RCT requirement.
    • “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3)
    • Relevant Quotes: 1) “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3) 2) “Within each school, one class was randomly selected to receive the CAL intervention, leaving the other class (classes) as control group(s).” (Fig. 1, p. 4) Detailed Analysis: The design randomises entire classes within each school, fulfilling the ERCT standard’s requirement that the unit of randomisation be the class (or stronger). Final sentence explaining if criterion C is met because class‑level randomisation was implemented.
  • E

    Exam-based Assessment

    • The tests were custom assemblies of items from exam books, not formal standardized exams.
    • “Some experts from the Center for Examination of Beijing helped us pick questions for the tests… The math test included 29–32 questions … The Chinese test included 30–35 questions.” (p. 36)
    • Relevant Quotes: 1) “Some experts from the Center for Examination of Beijing helped us pick questions for the tests from official examination books and exercise books. The math test included 29–32 questions… The Chinese test included 30–35 questions.” (p. 36) 2) “We used the scores of the students on the math and Chinese tests as our measures of student academic performance.” (p. 36) Detailed Analysis: Criterion E requires a formal, widely recognized standardized exam. Though experts contributed items, the assessments were custom-made and not an official standard exam. Final sentence: Criterion E is not met because the tests were custom instruments.
  • T

    Term Duration

    • Student performance was assessed at the end of the fall semester, meeting the term‑duration requirement.
    • “The third-round survey was a final evaluation survey conducted at the end of the program in late December…” (p. 7)
    • Relevant Quotes: 1) “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3) 2) “The third-round survey was a final evaluation survey conducted at the end of the program in late December, a time that coincided with the end of the Fall semester.” (p. 7) Detailed Analysis: The intervention and measurement span a full academic term (the Fall semester), ensuring outcomes are collected at least one term after the start. Final sentence explaining if criterion T is met because outcomes were measured after a full term.
  • D

    Documented Control Group

    • The control classes’ makeup, treatment conditions, and baseline data are clearly reported.
    • “Students in the classes that were assigned to the control group did not receive any CAL intervention.” (p. 4)
    • Relevant Quotes: 1) “Students in the classes that were assigned to the control group did not receive any CAL intervention… Following the protocol, they were not allowed to access the computers.” (p. 4) 2) Baseline and follow‑up data, including class sizes, demographics and attrition rates for control classes, are fully tabulated (Tables 1–2). Detailed Analysis: The control classes are described in detail: their treatment status, sample size, baseline characteristics, and attrition. This allows clear comparisons. Final sentence explaining if criterion D is met because the control group is comprehensively documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The trial randomised classes within schools rather than entire schools.
    • “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3)
    • Relevant Quotes: 1) “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3) Detailed Analysis: Randomisation was limited to classes within schools. No random assignment occurred at the school level. Final sentence explaining if criterion S is not met because only class‑level randomisation occurred.
  • I

    Independent Conduct

    • The authors who developed the CAL were also responsible for its implementation and assessment.
    • “Under the supervision of one teacher–supervisor trained by our research group…” (p. 4)
    • Relevant Quotes: 1) “Under the supervision of one teacher–supervisor trained by our research group, the students in the treatment group had two 40-min CAL sessions per week…” (p. 4) 2) “Our research team randomly selected one third grade class in each of the 24 program schools to receive the CAL intervention…” (p. 6) Detailed Analysis: The same team designed, implemented, and evaluated the intervention, with no independent third‑party evaluator noted. Final sentence explaining if criterion I is not met because conduct was not independent.
  • Y

    Year Duration

    • Tracking ceased at the semester’s end, not over a full academic year.
    • “The third-round survey… conducted at the end of the Fall semester.” (p. 7)
    • Relevant Quotes: 1) “We conducted a clustered (at the class level) RCT of CAL in Beijing migrant schools in the fall semester of 2010.” (p. 3) 2) “The third-round survey was a final evaluation survey conducted at the end of the program in late December…” (p. 7) Detailed Analysis: Outcomes are tracked only through a single semester; no year‑long follow‑up is reported. Final sentence explaining if criterion Y is not met because duration was less than one academic year.
  • B

    Balanced Resources

    • The additional CAL sessions are the treatment itself, so the control group’s business‑as‑usual status is appropriate.
    • “Under the supervision of one teacher–supervisor… the students… had two 40-min CAL sessions per week…” (p. 4)
    • Relevant Quotes: 1) “Under the supervision of one teacher–supervisor trained by our research group, the students in the treatment group had two 40-min CAL sessions per week during lunch break or after school…” (p. 4) 2) “Students in the classes that were assigned to the control group did not receive any CAL intervention. Following the protocol, they were not allowed to access the computers.” (p. 4) Detailed Analysis: The intervention’s extra instructional time via CAL is the primary treatment variable. The control group receives business-as-usual, consistent with the study’s intent to test the effect of those additional CAL resources. Final sentence explaining if criterion B is met because extra resources are integral to the tested intervention.

• Level 3 Criteria

  • R

    Reproduced Results

    • The paper contains no mention of independent replication by a different research team.
    • Relevant Papers: No independent replication of this trial is reported in peer‑reviewed journals by external teams. Detailed Analysis: The authors do not cite any studies where different researchers reproduced this exact experiment in another setting. Final sentence explaining if criterion R is not met because no replication is reported.
  • A

    All Exams

    • The study assessed only math and Chinese; other core subjects were omitted.
    • “In each round of survey… students were given a standardized math test and a standardized Chinese test.” (p. 6)
    • Relevant Quotes: 1) “In each round of survey… students were given a standardized math test and a standardized Chinese test.” (p. 6) Detailed Analysis: Only two subjects (math and Chinese) were tested. Core areas like science or social studies were not assessed. Final sentence explaining if criterion A is not met because not all main subjects were evaluated.
  • G

    Graduation Tracking

    • Student outcomes were not monitored beyond the semester, so no graduation tracking occurred.
    • Relevant Quotes: 1) “The third-round survey… conducted at the end of the Fall semester.” (p. 7) Detailed Analysis: No follow‑up beyond the fall semester is reported; students are not tracked to graduation. Final sentence explaining if criterion G is not met because graduation tracking is absent.
  • P

    Pre-Registered Protocol

    • There is no evidence the trial was pre-registered before data collection.
    • Relevant Quotes: No pre-registration statement or registry identifier appears in the document. Detailed Analysis: The study protocol is not referenced in any registry prior to data collection. Final sentence explaining if criterion P is not met because no pre-registration is mentioned.