Self-learning at the right level, COVID-19, school closure, and non-cognitive abilities

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The original RCT randomized at the school level, which satisfies the class-level (or stronger) RCT requirement.
    • We introduced the Kumon method in randomly selected BPSs for grades 3 and 4, covering 34 schools (17 treatment and 17 control schools) and approximately 1,000 students in total.
    • Relevant Quotes: 1) "We introduced the Kumon method in randomly selected BPSs for grades 3 and 4, covering 34 schools (17 treatment and 17 control schools) and approximately 1,000 students in total." (p. 2-3) Detailed Analysis: Criterion C requires an RCT conducted at the class level (or stronger). The paper explicitly states that the Kumon method was introduced in "randomly selected" BRAC Primary Schools (BPSs), with 34 schools split into 17 treatment and 17 control schools. Randomizing at the school level is stronger than class-level randomization and therefore satisfies Criterion C. Final Summary: Criterion C is met because randomization occurred at the school level, which is stronger than the required class-level design.
  • E

    Exam-based Assessment

    • The long-term follow-up cognitive outcome uses a custom rapid math test rather than a standardized exam.
    • Because we were unable to administer mathematics quizzes of the same type as the DT to comprehensively assess respondents’ cognitive abilities, we used rapid math tests as a proxy measure.
    • Relevant Quotes: 1) "Because we were unable to administer mathematics quizzes of the same type as the DT to comprehensively assess respondents’ cognitive abilities, we used rapid math tests as a proxy measure." (p. 5) 2) "The rapid assessment of cognitive abilities consists of seven math questions, which include slightly more difficult calculations than the original Kumon materials (see the Online Appendix E for the set of questions)." (p. 3) Detailed Analysis: Criterion E requires standardized exam-based assessments that are widely recognized, not assessments created or assembled for the study. The paper states that the follow-up could not use the original diagnostic test and instead used a "rapid math test" as a proxy, consisting of seven math questions with the item set provided in an online appendix. This indicates a study-specific assessment rather than a standardized exam. Final Summary: Criterion E is not met because the follow-up cognitive measure is a custom rapid math test (seven questions), not a standardized exam.
  • T

    Term Duration

    • The intervention ran for eight months, exceeding a full academic term.
    • In the treatment schools, students were asked to join daily 30 min self-learning Kumon sessions before regular lessons of BPSs over 8 months.
    • Relevant Quotes: 1) "In the treatment schools, students were asked to join daily 30 min self-learning Kumon sessions before regular lessons of BPSs over 8 months." (p. 3) Detailed Analysis: Criterion T requires outcome tracking at least one academic term after the intervention begins (typically about 3-4 months). The paper describes the intervention as running "over 8 months," which is clearly longer than a single term. Final Summary: Criterion T is met because the intervention duration of 8 months exceeds the minimum one-term requirement.
  • D

    Documented Control Group

    • The paper documents the control group with baseline and follow-up descriptive statistics in Table 1.
    • Table 1 shows the descriptive statistics and the raw comparison between the treatment and control groups using an unconditional t-test for baseline scores, household characteristics (household size and household head’s age and education), and ‘‘endline’’ outcomes from the Phase 2 survey in February 2022 (Fig. 1).
    • Relevant Quotes: 1) "Table 1 shows the descriptive statistics and the raw comparison between the treatment and control groups using an unconditional t-test for baseline scores, household characteristics (household size and household head’s age and education), and ‘‘endline’’ outcomes from the Phase 2 survey in February 2022 (Fig. 1)." (p. 3) 2) "DT score ... Control: 0.048" (Table 1, p. 4) 3) "Household size ... Control: 4.265" (Table 1, p. 4) Detailed Analysis: Criterion D requires that the control group be clearly documented, including baseline characteristics and outcomes. The paper explicitly points readers to Table 1 for descriptive statistics and raw comparisons between treatment and control groups, including baseline scores and household characteristics. Table 1 provides control-group values for multiple baseline measures (for example DT score and household size), satisfying the documentation requirement. Final Summary: Criterion D is met because the control group is documented with baseline and follow-up descriptive statistics in Table 1.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization is at the school level with 34 schools as clusters.
    • Clusters are schools at the baseline. There are 34 clusters.
    • Relevant Quotes: 1) "We introduced the Kumon method in randomly selected BPSs for grades 3 and 4, covering 34 schools (17 treatment and 17 control schools) and approximately 1,000 students in total." (p. 2-3) 2) "Clusters are schools at the baseline. There are 34 clusters." (Table 1 note, p. 4) Detailed Analysis: Criterion S requires an RCT conducted at the school level. The paper describes 34 schools split into treatment and control, and Table 1 notes that "Clusters are schools at the baseline," confirming school-level clustering consistent with school-level randomization. Final Summary: Criterion S is met because schools are the unit of randomization and clustering (34 schools).
  • I

    Independent Conduct

    • The evaluation team is not the Kumon organization and the paper declares no conflict of interest.
    • The authors declare that they have no conflict of interest related to this study.
    • Relevant Quotes: 1) "The authors declare that they have no conflict of interest related to this study." (p. 1) 2) "We also thank the authorities of the BRAC, Kumon Institute of Education Co., Ltd., and the Japan International Cooperation Agency (JICA) for cooperation in implementing the study." (p. 1) Detailed Analysis: Criterion I requires that the study be conducted independently of the intervention designers. The intervention is explicitly tied to "Kumon," a program associated with Kumon Institute of Education Co., Ltd. The authors acknowledge Kumon Institute of Education Co., Ltd. for "cooperation," which is consistent with Kumon being an external partner rather than the evaluator. The paper further declares no conflict of interest. Based on these statements, the evaluation appears to be conducted by researchers outside the Kumon organization. Final Summary: Criterion I is met because the authors are not the intervention provider and they explicitly declare no conflict of interest.
  • Y

    Year Duration

    • Outcomes are measured in a follow-up conducted six years after the original RCT period.
    • In 2022, we followed up students who joined an RCT study on self-learning at the right level from August 2015 to April 2016 (hereafter, the original RCT) after 6 years (Fig. 1).
    • Relevant Quotes: 1) "We revisit these students almost six years after the intervention followed by COVID-19 school closures." (p. 1) 2) "In 2022, we followed up students who joined an RCT study on self-learning at the right level from August 2015 to April 2016 (hereafter, the original RCT) after 6 years (Fig. 1)." (p. 2) Detailed Analysis: Criterion Y requires outcome measurement at least one academic year after the intervention begins. The paper states the original RCT ran from August 2015 to April 2016 and that the follow-up occurred "after 6 years" in 2022. This far exceeds the one-year minimum. Final Summary: Criterion Y is met because the follow-up occurs about six years after the original intervention period began.
  • B

    Balanced Control Group

    • The intervention adds time and materials, but these added resources are integral to the treatment being evaluated.
    • In the treatment schools, students were asked to join daily 30 min self-learning Kumon sessions before regular lessons of BPSs over 8 months.
    • Relevant Quotes: 1) "In the treatment schools, students were asked to join daily 30 min self-learning Kumon sessions before regular lessons of BPSs over 8 months." (p. 3) 2) "The intervention package included self-learning math worksheets, achievement tests, and record books to track progress." (p. 3) 3) "In particular, we investigate the long-term effects of Kumon’s ‘‘self-learning at the right level’’ program ..." (p. 1) Detailed Analysis: Criterion B requires balanced time and resources across treatment and control unless the additional resources are explicitly integral to what is being tested. Here, the treatment is the Kumon "self-learning at the right level" program, which is defined as adding daily 30-minute sessions plus program materials (worksheets, achievement tests, record books). The paper frames the study goal as investigating the effects of this Kumon program. Because the extra time and materials are core components of the program itself (not an unrelated add-on), the study is explicitly evaluating the impact of providing this supplementary program relative to business as usual. Final Summary: Criterion B is met because the additional time and materials are integral components of the intervention being tested.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication by a different research team was found for this specific Kumon RCT.
    • Relevant Quotes: 1) "There remain two important tasks for further investigations. First, it is imperative to test the external validity of the above findings, ad- dressing the caveats of attrition and the resulting selection bias." (p. 7) Detailed Analysis: Criterion R requires an independent replication by a different research team, in a different context, published in a peer-reviewed journal. The paper itself does not report that an independent team replicated this specific Kumon RCT. In addition, an internet search focused on replications of the Bangladesh BRAC-school Kumon RCT did not identify a peer-reviewed reproduction study by different authors that clearly replicated the same intervention and design. Final Summary: Criterion R is not met because no independent, peer-reviewed replication of this specific RCT by a different research team was found.
  • A

    All-subject Exams

    • Only mathematics was assessed (and not via standardized exams), so the study does not provide all-subject standardized exam outcomes.
    • The rapid assessment of cognitive abilities consists of seven math questions, which include slightly more difficult calculations than the original Kumon materials (see the Online Appendix E for the set of questions).
    • Relevant Quotes: 1) "The questionnaire contained questions on the learning status, a rapid assessment of cognitive abilities, non-cognitive survey questions, and questions on the risk preference, patience, and peer effect." (p. 3) 2) "The rapid assessment of cognitive abilities consists of seven math questions, which include slightly more difficult calculations than the original Kumon materials (see the Online Appendix E for the set of questions)." (p. 3) Detailed Analysis: Criterion A requires standardized exam-based measurement across all main subjects. This paper’s cognitive assessment in the follow-up is limited to mathematics (seven math questions) and does not report standardized exam outcomes across core subjects. Additionally, Criterion E is not met (the follow-up uses a custom rapid math test rather than a standardized exam), and by the ERCT rule, if E is not met then A cannot be met. Final Summary: Criterion A is not met because the study does not provide standardized exam outcomes across all core subjects.
  • G

    Graduation Tracking

    • The paper does not show systematic tracking of participants through graduation for the full cohort.
    • The percentages do not add up to 100 because some of the students had graduated.
    • Relevant Quotes: 1) "In 2022, we followed up students who joined an RCT study on self-learning at the right level from August 2015 to April 2016 (hereafter, the original RCT) after 6 years (Fig. 1)." (p. 2) 2) "The percentages do not add up to 100 because some of the students had graduated." (Table 1 note, p. 4) Detailed Analysis: Criterion G requires tracking participants until graduation. The paper reports a follow-up survey conducted about six years after the original RCT period and notes that "some of the students had graduated," but it does not describe a design where the cohort is tracked through graduation as an explicit endpoint for all participants. Instead, graduation appears as a status outcome observed at the time of the follow-up survey, alongside outcomes such as current attendance and dropout. Therefore, the paper does not provide evidence that participants were systematically followed until graduation for the whole cohort. Final Summary: Criterion G is not met because the study does not demonstrate cohort-wide tracking through graduation as a defined endpoint.
  • P

    Pre-Registered

    • The paper cites an AEA RCT Registry ID but does not provide (and we could not verify) a registration date before the study start.
    • The original study (Sawada et al. 2024) has been registered at the American Economic Association’s Randomized Controlled Trials (RCTs) registry (AEARCTR-0002925).
    • Relevant Quotes: 1) "The original study (Sawada et al. 2024) has been registered at the American Economic Association’s Randomized Controlled Trials (RCTs) registry (AEARCTR-0002925)." (p. 1) 2) "In 2022, we followed up students who joined an RCT study on self-learning at the right level from August 2015 to April 2016 (hereafter, the original RCT) ..." (p. 2) Detailed Analysis: Criterion P requires that the protocol be pre-registered before the study begins. The paper provides a registry identifier (AEARCTR-0002925) but does not report the registry posting date. A targeted attempt to verify the registry entry details (including the registration date) using publicly accessible online sources did not yield a verifiable registration-date record for this specific ID. Without evidence that the registration date precedes the study start (August 2015), the pre-registration requirement is not satisfied under the ERCT standard. Final Summary: Criterion P is not met because a pre-intervention registration date could not be verified from the registry or the paper.