The heterogeneous effect of information on student performance: Evidence from a randomized control trial in Mexico

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation occurred at the school level, which meets or exceeds the class‑level RCT requirement.
    • "Following a two-step stratified sampling by regions (north, center, and south), the 54 schools were randomly divided into 26 treatment schools and 28 control schools."
    • Relevant Quotes: 1) "Following a two-step stratified sampling by regions (north, center, and south), the 54 schools were randomly divided into 26 treatment schools and 28 control schools." (p. 4) Detailed Analysis: The study clearly states that entire schools were the unit of randomisation. School-level randomisation exceeds the requirement for class‑level RCT, satisfying the criterion by design. Final sentence: The study randomizes at the school level, meeting the Class‑level RCT criterion.
  • E

    Exam-based Assessment

    • The study uses the nationally standardized ENLACE exam for objective, comparable assessment.
    • "Using 2012 and 2013 administrative data from the 12th grade census-based nationally standardized ENLACE exam and the university entry exam EXANI‑II ..."
    • Relevant Quotes: 1) "Using 2012 and 2013 administrative data from the 12th grade census-based nationally standardized ENLACE exam and the university entry exam EXANI‑II ..." (p. 2) Detailed Analysis: The authors employ ENLACE, a national standardized test, as the main outcome measure. This exam is widely recognised and comparable across contexts, satisfying the requirement for a standard exam‑based assessment. Final sentence: The use of the ENLACE standardized exam meets the Exam‑based Assessment criterion.
  • T

    Term Duration

    • Outcomes were collected approximately three years after the intervention began, exceeding the minimum of one academic term.
    • "In November 2009, the baseline data was collected and the information treatment was delivered. Using 2012 and 2013 administrative data from the 12th grade ... we measure the impact of the information treatment ..."
    • Relevant Quotes: 1) "In November 2009, the baseline data was collected and the information treatment was delivered." (p. 2) 2) "Our results show that, almost three years after the treatment was implemented, ... had a sizeable and statistically significant effect on the ENLACE test score—0.22 standard deviations (σ)." (p. 2) Detailed Analysis: The intervention began in late 2009 and primary outcomes were measured using data from 2012, nearly three academic years later. This far exceeds one full term between the start of the intervention and the outcome measurement. Final sentence: The follow-up spanned multiple terms after the intervention, so Term Duration is met.
  • D

    Documented Control Group

    • Baseline demographics and pre‑intervention scores for the control group are provided in detail.
    • "Table 1 shows the baseline characteristics for the full sample as well as separately for boys and girls, distinguishing between students in the treatment and control groups."
    • Relevant Quotes: 1) "Table 1 shows the baseline characteristics for the full sample as well as separately for boys and girls, distinguishing between students in the treatment and control groups." (p. 6) Detailed Analysis: The authors present detailed demographic and baseline test‑score data for both treatment and control groups in Table 1. This table clearly documents the control group’s composition and baseline metrics, fulfilling the documentation requirement. Final sentence: The control group’s baseline characteristics are clearly documented, so this criterion is met.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation at the school level fulfills the School‑level RCT requirement.
    • "Following a two-step stratified sampling by regions (north, center, and south), the 54 schools were randomly divided into 26 treatment schools and 28 control schools."
    • Relevant Quotes: 1) "Following a two-step stratified sampling by regions (north, center, and south), the 54 schools were randomly divided into 26 treatment schools and 28 control schools." (p. 4) Detailed Analysis: The paper explicitly randomises entire schools rather than individual classes or students. This satisfies the stronger requirement for a school‑level RCT. Final sentence: School‑level randomisation meets the School‑level RCT criterion.
  • I

    Independent Conduct

    • The evaluation was conducted by researchers unaffiliated with SEP, ensuring independence of the study’s implementation and analysis.
    • "We … are especially indebted to Martha Hernandez, Elizabeth Monroy, and Paula Villasenor, who were responsible for project and data management at SEP … The views expressed here are those of the authors alone."
    • Relevant Quotes: 1) "In 2009 the Mexican Secretariat of Public Education (SEP) ... designed and piloted an intervention ... known as Percepciones, the pilot program included an evaluation strategy based on a stratified randomized control trial." (pp. 319–320) 2) "We … are especially indebted to Martha Hernandez, Elizabeth Monroy, and Paula Villasenor, who were responsible for project and data management at SEP … The views expressed here are those of the authors alone." (Acknowledgments) Detailed Analysis: The Percepciones pilot was designed and implemented by SEP, while the data collection, analysis, and reporting were carried out by external researchers affiliated with the World Bank and University of Surrey. This clear separation of design and evaluation ensures an independent conduct of the trial. Final sentence: An external research team conducted the evaluation, fulfilling the Independent Conduct criterion.
  • Y

    Year Duration

    • Participants were followed for approximately three years, which is longer than one academic year from intervention start to final outcomes.
    • "Fig. 1 shows the timeline of the project spanning from May 2009 to May 2012."
    • Relevant Quotes: 1) "Fig. 1 shows the timeline of the project spanning from May 2009 to May 2012." (p. 4) Detailed Analysis: The experimental period covers nearly three years, from mid-2009 to mid-2012, encompassing multiple academic years. The outcomes were evaluated well over a year after the intervention began, satisfying the Year Duration criterion. Final sentence: The study’s timeline extends past one full academic year, so Year Duration is met.
  • B

    Balanced Resources

    • The difference in time or resources between groups is trivial and integral to the intervention, unlikely to bias the results.
    • "On average, students in the treatment group spent 12 min interacting with the interface."
    • Relevant Quotes: 1) "On average, students in the treatment group spent 12 min interacting with the interface." (p. 4) Detailed Analysis: Although the treatment group received a brief 12‑minute information session, this additional time is minimal relative to total class time. The control group continued with business-as-usual schooling and was not deprived of any substantial instructional time or resources. Final sentence: The minimal extra time in the intervention is negligible, so Balanced Resources is met.

• Level 3 Criteria

  • R

    Reproduced Results

    • The paper does not reference any independent replication studies, and none were found in external literature.
    • Relevant Quotes: 1) *No mention of any independent replication study is found in the paper.* Detailed Analysis: The authors do not cite any separate research team that attempted to replicate this intervention and its findings. A search for external replication studies did not reveal any published independent replications of this RCT. Final sentence: No independent replication is reported, so Reproduced is not met.
  • A

    All Exams

    • Only math and language outcomes are measured, omitting other main subjects.
    • "... on standardized test scores in math and language (Spanish) at the end of high school, ..."
    • Relevant Quotes: 1) "... on standardized test scores in math and language (Spanish) at the end of high school, ..." (p. 2) Detailed Analysis: The study’s academic outcomes are limited to math and Spanish test scores. Other core subjects (e.g., science or social studies) were not assessed, failing to meet the All‑subject Exams requirement. Final sentence: Only two subjects are assessed without justification, so this criterion is not met.
  • G

    Graduation Tracking

    • The RCT measured on-time high school completion, meaning participants were followed through the completion of that educational level.
    • "The Percepciones pilot displayed no impact on the probability of on-time high school graduation, but had a large positive effect on learning outcomes ..."
    • Relevant Quotes: 1) "The Percepciones pilot displayed no impact on the probability of on-time high school graduation, but had a large positive effect on learning outcomes ..." (p. 9) Detailed Analysis: Students were tracked through the end of high school to determine whether they graduated on time. The inclusion of on-time high school completion as an outcome indicates that the study followed participants to a key educational milestone. Final sentence: The study tracked students until high school graduation, so Graduation Tracking is met.
  • P

    Pre-Registered Protocol

    • No pre-registration statement or registry ID is provided in the paper or its references.
    • Relevant Quotes: 1) *No mention of pre-registration or any trial registry is found in the paper.* Detailed Analysis: The paper does not reference a pre-registered analysis plan or registration on a trial registry platform. The study appears to have been conducted without prior registration. Final sentence: Without evidence of pre-registration, this criterion is not met.