Disrupting Education? Experimental Evidence on Technology‑Aided Instruction in India

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the individual‑student level rather than at the class level, so the Class‑level RCT criterion is not satisfied.
    • "The 619 participants were individually randomized into treatment and control groups with 305 students in the control and 314 in the treatment group."
    • Relevant Quotes: 1) "The 619 participants were individually randomized into treatment and control groups with 305 students in the control and 314 in the treatment group." (p. 9) 2) "Randomization was stratified by center‑batch preferences." (p. 9) Detailed Analysis: These quotes make clear that randomisation occurred at the individual‑student level rather than at the class level. The ERCT Standard’s Class‑level RCT criterion requires entire classes (or schools) to be randomized to prevent contamination. Because students within the same cohort were assigned individually, this criterion is not met. Therefore, criterion C is not met because randomisation was conducted at the individual‑student level rather than at the class level.
  • E

    Exam-based Assessment

    • The study used researcher‑designed custom tests rather than a standardized, widely recognized exam, so the Exam‑based Assessment criterion is not satisfied.
    • "The tests were designed independently by the research team and intended to capture a wide range of student achievement."
    • Relevant Quotes: 1) "The tests were designed independently by the research team and intended to capture a wide range of student achievement." (p. 11) 2) "Test items ranged in difficulty from 'very easy' questions ... to 'grade‑appropriate' competencies found in international assessments." (p. 11) Detailed Analysis: The assessment instruments were custom‑designed by the authors. The ERCT Exam‑based Assessment criterion requires use of a standardized, widely recognized exam. Since bespoke tests were used, this criterion is not met. Therefore, criterion E is not met because the study used custom‑designed assessments instead of a widely recognized standardized exam.
  • T

    Term Duration

    • Outcomes were measured after a 4.5‑month intervention period, which covers at least one term, satisfying the Term Duration criterion.
    • "We measure program impacts using ... tests ... before and after the 4.5‑month‑long intervention."
    • Relevant Quotes: 1) "We measure program impacts using ... tests ... before and after the 4.5‑month‑long intervention." (p. 3) 2) "Baseline assessments in September 2015 ... endline in February 2016." (p. 5) Detailed Analysis: The intervention spanned approximately 4.5 months, exceeding a typical academic term of 3–4 months. Under the ERCT Term Duration criterion, measuring outcomes after at least one full term satisfies the requirement. Therefore, criterion T is met because outcomes were measured after a 4.5‑month period, exceeding a full academic term.
  • D

    Documented Control Group

    • The study provides detailed baseline characteristics and assessment outcomes for the control group, fulfilling the Documented Control Group criterion.
    • "The treatment and control groups did not differ significantly at baseline on gender, socioeconomic status (SES), or baseline test scores (Table 1, panel A)."
    • Relevant Quotes: 1) "The treatment and control groups did not differ significantly at baseline on gender, socioeconomic status (SES), or baseline test scores (Table 1, panel A)." (p. 9) 2) "Students not chosen by lottery ... completed an endline assessment." (p. 5) Detailed Analysis: Table 1 and its narrative describe the control group’s composition, demographics, and baseline performance. This clear documentation meets the Documented Control Group criterion. Therefore, criterion D is met because the control group’s characteristics and baseline performance are comprehensively documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was done at the individual‑student level, not at the school level, so the School‑level RCT criterion is not satisfied.
    • "The 619 participants were individually randomized into treatment and control groups."
    • Relevant Quotes: 1) "The 619 participants were individually randomized ..." (p. 9) 2) "Randomization was stratified by center‑batch preferences." (p. 9) Detailed Analysis: Randomisation occurred at the individual‑student level, not at the school level. The School‑level RCT criterion requires entire schools to be randomized. Hence this criterion is not met. Therefore, criterion S is not met because randomisation was not conducted at the school level.
  • I

    Independent Conduct

    • The same team that designed Mindspark also carried out the trial and analysis, so the Independent Conduct criterion is not satisfied.
    • "Developed by ... Educational Initiatives (EI), the Mindspark software reflects over a decade of iterative product development and was evaluated by the same team."
    • Relevant Quotes: 1) "Developed by ... Educational Initiatives (EI), the Mindspark software reflects ... field support ... by the same team." (p. 7) 2) "At the demonstration sessions, ... by EI staff." (p. 9) Detailed Analysis: The creators of the intervention also conducted the evaluation, with no independent third‑party oversight. Under the Independent Conduct criterion, this constitutes a failure. Therefore, criterion I is not met because the evaluation was not conducted independently of the intervention’s developers.
  • Y

    Year Duration

    • Participants were followed for only 4.5 months rather than an academic year, so the Year Duration criterion is not satisfied.
    • "We measure program impacts ... after the 4.5‑month‑long intervention."
    • Relevant Quotes: 1) "We measure program impacts ... after the 4.5‑month‑long intervention." (p. 3) 2) No follow‑up beyond February 2016 is reported. Detailed Analysis: Tracking lasted only 4.5 months, far short of a full academic year. The Year Duration criterion thus is not met. Therefore, criterion Y is not met because participants were only tracked for 4.5 months, not a full academic year.
  • B

    Balanced Resources

    • The intervention’s extra instructional time is integral to the treatment, so the Balanced Resources criterion is satisfied.
    • "The centers scheduled 6 days of instruction per week, for 90 minutes per day."
    • Relevant Quotes: 1) "The centers scheduled 6 days of instruction per week, for 90 minutes per day." (p. 7) 2) "Students not chosen by lottery ... did not have access to the Mindspark program." (p. 9) Detailed Analysis: The additional instructional time is the core treatment being tested. Under the Balanced Resources criterion, when extra resources are integral to the intervention, the control may remain business‑as‑usual. Here, the extra time is part of the treatment design, so the criterion is met. Therefore, criterion B is met because the extra instructional time is integral to the intervention.

• Level 3 Criteria

  • R

    Reproduced Results

    • No independent replication of the study is reported, so the Reproduced criterion is not satisfied.
    • Relevant Quotes: None found regarding independent replication. Detailed Analysis: No external replication is reported in the paper. Therefore, the Reproduced criterion is not met. Therefore, criterion R is not met because no independent replication of the study has been documented.
  • A

    All Exams

    • Only mathematics and Hindi were assessed, so the All‑subject Exams criterion is not satisfied.
    • "We measure program impacts using ... tests of student learning in math and Hindi."
    • Relevant Quotes: 1) "We measure program impacts using ... tests of student learning in math and Hindi." (p. 3) 2) "Scheduled daily instruction ... provided activities on math, Hindi, and English one day a week." (p. 7) Detailed Analysis: Assessment was limited to mathematics and Hindi; core subjects beyond these were not evaluated. The All‑subject Exams criterion thus is not met. Therefore, criterion A is not met because only two core subjects were assessed.
  • G

    Graduation Tracking

    • Participants were only followed until the endline test, with no graduation tracking, so the Graduation Tracking criterion is not satisfied.
    • Relevant Quotes: 1) "Baseline assessments in September 2015 ... endline in February 2016." (p. 5) 2) No further follow‑up is described. Detailed Analysis: The study ends at the endline test and does not track participants through graduation. Thus, the Graduation Tracking criterion is not met. Therefore, criterion G is not met because participants were not tracked through graduation.
  • P

    Pre-Registered Protocol

    • The trial was registered only after data collection began, so the Pre‑registered Protocol criterion is not satisfied.
    • "The study was registered with the AEA Trial Registry (RCT ID AEARCTR‑0000980)."
    • Relevant Quotes: 1) "The study was registered with the AEA Trial Registry (RCT ID AEARCTR‑0000980)." (p. 1 footnote) Detailed Analysis: The study was registered in the AEA trial registry on April 27, 2016, which was after the start of the experiment in 2015. Because registration occurred only after data collection had begun, the Pre‑registered Protocol criterion is not met. Therefore, criterion P is not met because the trial was registered only after data collection had started.