Integrating learning platforms within regular school time: experimental evidence from Chilean primary schools

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized at the section (class) level within schools, which meets the requirement for class-level randomization.
    • "We randomly assigned one fourth-grade section within each of these schools to the treatment group and assigned the other section in that grade to the control group."
    • Relevant Quotes: 1) "We randomly assigned one fourth-grade section within each of these schools to the treatment group and assigned the other section in that [cite_start]grade to the control group." [cite: 59] 2) "We adopted a within-school, section-level randomization design." [cite_start][cite: 244] Detailed Analysis: The criterion requires that randomization be conducted at the class level (or higher, i.e., school level) to prevent contamination within the classroom. The authors explicitly state that they randomized "sections" (classrooms) within each school, assigning one to treatment and one to [cite_start]control[cite: 59, 244]. This ensures that the unit of randomization is the class, fulfilling the criterion. Final sentence explaining if criterion C is met because randomization was performed at the section (class) level.
  • E

    Exam-based Assessment

    • The study used the SIMCE, which is the Chilean national standardized exam.
    • "Our primary outcome measuring math learning was obtained from the Chilean national standardized exam administered in November 2017"
    • Relevant Quotes: 1) "Our primary outcome measuring math learning was obtained from the Chilean national standardized exam administered in November 2017" [cite_start][cite: 63] 2) "This is a paper-based assessment implemented yearly in all schools in the country to monitor math and language learning in fourth grade." [cite_start][cite: 64] Detailed Analysis: The criterion requires the use of widely recognized standardized exams rather than custom-made tests. The study utilizes the SIMCE ("Sistema de Medición de la Calidad de la Educación"), which is the national [cite_start]standardized exam in Chile[cite: 63, 233]. This explicitly meets the requirement for a standard, widely recognized test. Final sentence explaining if criterion E is met because the primary outcome was measured using the national standardized SIMCE exam.
  • T

    Term Duration

    • The intervention lasted approximately seven months, which exceeds the minimum one-term duration requirement.
    • "The program was implemented immediately thereafter and ran until November 2017... (after seven months of program exposure)."
    • Relevant Quotes: 1) "Baseline data were collected in March 2017; the program was implemented immediately thereafter and ran until November 2017" [cite_start][cite: 61] 2) "Our primary outcome measuring math learning was obtained from the Chilean national standardized exam administered in November 2017 (after [cite_start]seven months of program exposure)." [cite: 63] Detailed Analysis: The criterion requires outcomes to be measured at least one full academic term (typically 3-4 months) after the intervention begins. This study [cite_start]ran from March to November 2017, a duration of seven months[cite: 61, 63]. This period is significantly longer than one academic term. Final sentence explaining if criterion T is met because the intervention duration of seven months exceeds the one-term minimum.
  • D

    Documented Control Group

    • The control group is clearly documented with demographic data and a description of their "business as usual" condition.
    • "The other sections were assigned to the control group, which did not have access to sessions with the learning platform and received traditional math instruction."
    • Relevant Quotes: 1) "The other sections were assigned to the control group, which did not have access to sessions with the learning platform and received [cite_start]traditional math instruction." [cite: 245] 2) "Table 2... presents descriptive statistics for baseline characteristics for the treatment and control groups... Treatment and [cite_start]control groups are well balanced at baseline." [cite: 318-319] 3) "47 % of students in the control group are girls, 99% attended [cite_start]kindergarten..." [cite: 321] Detailed Analysis: The criterion requires detailed documentation of the control group's characteristics and conditions. The paper provides a specific description of the control condition ("traditional math instruction" with no platform [cite_start]access)[cite: 245]. Furthermore, Table 2 and the text provide explicit demographic statistics for the control group (gender, kindergarten [cite_start]attendance, etc.) and confirm balance[cite: 318, 321]. Final sentence explaining if criterion D is met because the control group conditions and baseline characteristics are thoroughly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was performed at the section (class) level within schools, not at the school level.
    • "We adopted a within-school, section-level randomization design. Within each of the 24 participating schools, we randomly assigned one of the two fourth-grade sections to the treatment group."
    • Relevant Quotes: 1) "We adopted a within-school, section-level randomization design. Within each of the 24 participating schools, we randomly assigned one of [cite_start]the two fourth-grade sections to the treatment group." [cite: 244] 2) "We randomly assigned one fourth-grade section within each of these [cite_start]schools to the treatment group..." [cite: 59] Detailed Analysis: The criterion requires randomization to occur at the school level (among schools) to reflect real-world implementation and avoid spillover. This study explicitly utilized a "within-school" design where randomization [cite_start]occurred at the section (class) level, not the school level[cite: 244]. Therefore, the stronger school-level requirement is not met. Final sentence explaining if criterion S is not met because the study used class-level randomization within schools rather than randomizing entire schools.
  • I

    Independent Conduct

    • The study was not conducted independently; the lead author developed the program and the author team managed the implementation.
    • "The program employed a learning platform called Conectaldeas developed by a team at the Center for Advanced Research on Education of the Universidad de Chile."
    • Relevant Quotes: 1) "The program employed a learning platform called Conectaldeas developed by a team at the Center for Advanced Research on Education of [cite_start]the Universidad de Chile." [cite: 161] 2) "Roberto Araya, Elena Arias Ortiz, Nicolas Bottan, Julian Cristia... [cite_start]Institute of Education, Universidad de Chile" [cite: 8-9] 3) "Roberto Araya thanks... funds... and is grateful for initial funding... [cite_start]that supported development of the program." [cite: 40] Detailed Analysis: The criterion requires the study to be conducted independently from the designers of the intervention. The first author, Roberto Araya, is affiliated with the institution that developed the platform and is [cite_start]thanked for funding regarding the "development of the program"[cite: 40, 161]. The paper does not state that the evaluation was conducted by an independent third party; rather, the authors themselves conducted the study. Final sentence explaining if criterion I is not met because the lead author was involved in the development of the intervention being tested.
  • Y

    Year Duration

    • The intervention duration was seven months, which is less than the full academic year (typically 9-10 months) required by the criterion.
    • "Our primary outcome measuring math learning was obtained... after seven months of program exposure"
    • Relevant Quotes: 1) "The program ran from March to November 2017 (the school year in [cite_start]Chile takes place from March to December)." [cite: 61] 2) "Our primary outcome measuring math learning was obtained from the Chilean national standardized exam administered in November 2017 (after [cite_start]seven months of program exposure)." [cite: 63] Detailed Analysis: The criterion requires outcomes to be measured at least one full academic year (typically 9-10 months) after the start. The study explicitly states the school year runs from March to December, but the program and measurement concluded in November after "seven months of program [cite_start]exposure"[cite: 61, 63]. Seven months falls short of the full academic year requirement. Final sentence explaining if criterion Y is not met because the study duration was seven months, which is shorter than a full academic year.
  • B

    Balanced Control Group

    • The extra time and resources were an integral part of the "bundled" intervention explicitly being tested against business-as-usual, satisfying the exception for this criterion.
    • "Because of the bundled nature of the program, we cannot assess the contribution of each feature on the overall effect reported."
    • Relevant Quotes: 1) "The program evaluated packs several features including: ... (iii) 90 min of additional math instruction... (ii) sessions conducted by external [cite_start]coordinators" [cite: 55] 2) "Because of the bundled nature of the program, we cannot assess the contribution of each feature on the overall effect reported. Still, identifying effective packages... is relevant for policy purposes." [cite_start][cite: 56-57] 3) "The control group... did not have access to sessions with the [cite_start]learning platform and received traditional math instruction." [cite: 245] Detailed Analysis: The intervention group received significant additional resources, specifically 90 minutes of extra math instruction per week and an [cite_start]external coordinator[cite: 55], while the control group received only [cite_start]traditional instruction[cite: 245]. Normally, this imbalance would fail the criterion. However, the authors explicitly define the intervention as a "bundled program" and state that the intent is to evaluate this package [cite_start][cite: 55-57]. Under the ERCT standard exception, if the additional resources are the primary treatment variable (part of the bundle), the control group may remain "business as usual." Final sentence explaining if criterion B is met because the extra instructional time was an integral part of the bundled intervention being tested.

• Level 3 Criteria

  • R

    Reproduced

    • There is no evidence provided of an independent replication of this study by a different research team.
    • Relevant Quotes: 1) "We also study the early implementation of Conectaldeas... from 2011 [cite_start]until 2016... non-experimental estimates." [cite: 530] 2) "Araya (2018) evaluated Conectaldeas using a before and after [cite_start]approach..." [cite: 540] Detailed Analysis: The criterion requires the study to be independently replicated by a different research team. While the paper discusses previous implementations (2011-2016), these were analyzed by the same authors or [cite_start]related teams (Araya)[cite: 530, 540]. There is no citation or mention of an independent reproduction of the study results by an external group in the provided text. Final sentence explaining if criterion R is not met because no independent replication by a different research team is cited.
  • A

    All-subject Exams

    • The study assessed Math and Language but did not assess Science, which is stated as a subject taught by the teachers.
    • "Each teacher is assigned one classroom and they teach all academic subjects such as Math, Language, and Science."
    • Relevant Quotes: 1) "Each teacher is assigned one classroom and they teach all academic [cite_start]subjects such as Math, Language, and Science." [cite: 156] 2) "Our primary outcome measuring math learning was obtained from the Chilean national standardized exam... Effects on language were also [cite_start]analyzed to explore potential spillovers." [cite: 305] Detailed Analysis: The criterion requires the study to measure the impact on all main subjects taught. The paper explicitly states that teachers teach "Math, [cite_start]Language, and Science"[cite: 156]. However, the study only measures [cite_start]outcomes for Math and Language[cite: 305]. By excluding Science, the study fails to assess potential negative impacts on all main subjects. Final sentence explaining if criterion A is not met because Science, a main subject taught, was not assessed.
  • G

    Graduation Tracking

    • The study tracked students only until the end of the intervention period (Grade 4), not until graduation.
    • "The RCT ended at that point." (implied by lack of follow-up description)
    • Relevant Quotes: 1) "Our primary outcome... was obtained from the Chilean national [cite_start]standardized exam administered in November 2017" [cite: 63] 2) "There were no significant spillover effects... [analysis ends with [cite_start]exam data]" [cite: 521] Detailed Analysis: The criterion requires tracking participants until their graduation to assess long-term impacts. The provided text indicates that measurement [cite_start]occurred at the end of the program (November 2017)[cite: 63], with no mention of tracking the students through the end of their primary education or graduation. Final sentence explaining if criterion G is not met because there is no evidence of student tracking beyond the immediate intervention year.
  • P

    Pre-Registered

    • There is no evidence in the text that the study protocol was pre-registered before data collection began.
    • Relevant Quotes: 1) "This project was funded by the IDRC... and the IADB. Cornell IRB#0,147,331." [cite_start][cite: 43] [cite_start]2) "Baseline data were collected in March 2017..." [cite: 61] Detailed Analysis: The criterion requires the full study protocol to be pre-registered before data collection. [cite_start]The paper mentions funding and IRB approval[cite: 43], but there is no mention of a pre-registration on a public registry (like AEA RCT Registry or ClinicalTrials.gov) or a registration date prior to [cite_start]the March 2017 start date[cite: 61]. Final sentence explaining if criterion P is not met because no pre-registration details are provided.