Learning at the Last Mile: Evidence from a Randomized Controlled Trial of Computer-Assisted Instruction in Remote Schools of the Philippines

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized assignment at the school level, which satisfies and exceeds the class-level requirement.
    • Within each pair, one school was randomly assigned to participate in the program ('treatment' schools) while the other was assigned to the 'control' school group
    • Relevant Quotes: 1) "We identified 42 eligible schools for the tablet intervention pilot using a list of 'last mile schools'..." (p. 10) 2) "At the baseline, schools were paired within divisions based on similar sizes. Within each pair, one school was randomly assigned to participate in the program ('treatment' schools) while the other was assigned to the 'control' school group, so that there were 21 treatment and 21 control schools." (p. 10) Detailed Analysis: The study explicitly describes randomizing at the school level ("one school was randomly assigned..."). According to the ERCT standard, a school-level RCT is stronger than a class-level RCT and automatically satisfies the class-level criterion. The randomization was not done at the individual student level within classes, thus avoiding within-class contamination. Criterion C is met because the study randomized entire schools to treatment and control conditions.
  • E

    Exam-based Assessment

    • The study employed custom-developed 21-item assessments aligned with the curriculum rather than widely recognized standardized exams.
    • The assessments were validated by a panel of educators and DepEd officials to ensure alignment with target competencies and clarity. A pilot study of 60 students from Metro Manila further refined the tests
    • Relevant Quotes: 1) "Baseline and endline proficiency in math and English were measured using short assessments administered in person at the participating schools... The 21-item, 30-minute tests were aligned with DepEd's MELCs and included concepts and contents from up to three lower grade levels." (p. 11) 2) "The assessments were validated by a panel of educators and DepEd officials to ensure alignment with target competencies and clarity. A pilot study of 60 students from Metro Manila further refined the tests..." (p. 11) Detailed Analysis: The ERCT standard requires the use of widely recognized standardized exams (like national exams or established international assessments). While the tests used here were "aligned with DepEd's MELCs" (Minimum Expected Learning Competencies) and validated by a panel, they were specifically developed and refined ("refined the tests", "final design") for this study rather than being an off-the-shelf standardized instrument. They are described as "short assessments" administered specifically for the data collection. This falls under the category of custom-made assessments. Criterion E is not met because the study used custom-designed assessments validated for the project rather than widely recognized standardized exams.
  • T

    Term Duration

    • The study tracked outcomes from August 2022 to May 2024, covering almost two full academic years.
    • The endline survey was administered in May 2024, marking the completion of the second academic year during which students had access to the tablets.
    • Relevant Quotes: 1) "In August 2022, all students in the treatment schools received a tablet..." (p. 10) 2) "The endline survey was administered in May 2024, marking the completion of the second academic year during which students had access to the tablets." (p. 12) Detailed Analysis: The intervention began with tablet distribution in August 2022. Outcome measurements (endline survey) were collected in May 2024. This duration spans approximately 21 months (almost two full academic years). The criterion requires a minimum duration of one full academic term (~3-4 months). Since the interval is nearly two years, it far exceeds the minimum requirement. Criterion T is met because the intervention and tracking period lasted significantly longer than one academic term.
  • D

    Documented Control Group

    • The control group's baseline characteristics, including demographics and test scores, are fully documented and compared in Table 1.
    • Table 1 also checks for baseline balance across 11 parental, household, and student characteristics. All of these parental, household and student characteristics are balanced at baseline
    • Relevant Quotes: 1) "Within each pair, one school was randomly assigned to participate in the program ('treatment' schools) while the other was assigned to the 'control' school group..." (p. 10) 2) "Table 1 also checks for baseline balance across 11 parental, household, and student characteristics. All of these parental, household and student characteristics are balanced at baseline..." (p. 11) 3) "Control" column in Table 1 lists specific demographics (Mother's education, devices, electricity, etc.) and baseline test scores. (p. 16) Detailed Analysis: The paper provides a clear documentation of the control group. Table 1 explicitly lists the characteristics of the control group ("Control" column) including sample size (N=264 at baseline), mother's education, asset ownership, and baseline test scores. The text confirms that control schools were paired based on size and that baseline characteristics were balanced. The control condition is described as the prevailing context (paper-based modules/limited face-to-face) without the tablet intervention. Criterion D is met because the characteristics and conditions of the control group are well-documented and presented in comparison tables.

• Level 2 Criteria

  • S

    School-level RCT

    • The study randomized 42 schools into treatment and control groups, satisfying the school-level randomization criterion.
    • Within each pair, one school was randomly assigned to participate in the program ('treatment' schools) while the other was assigned to the 'control' school group
    • Relevant Quotes: 1) "At the baseline, schools were paired within divisions based on similar sizes. Within each pair, one school was randomly assigned to participate in the program ('treatment' schools) while the other was assigned to the 'control' school group..." (p. 10) 2) "Because the treatment was assigned at the school level... we carry out regression analyses accounting for group-level dependencies by clustering standard errors at the school level..." (p. 12) Detailed Analysis: The study randomly assigned the intervention at the school level (21 treatment vs 21 control schools). This meets the requirement for a School-level RCT, which is designed to prevent contamination and reflect real-world implementation units. The analysis also accounts for this clustering. Criterion S is met because the unit of randomization was explicitly the school.
  • I

    Independent Conduct

    • The study was conducted by authors affiliated with the Asian Development Bank, which also funded and supported the implementation of the intervention.
    • The study also would not have been possible without the implementation efforts of ADB colleagues...
    • Relevant Quotes: 1) "Together with the Department of Education, we conducted a 42-school randomized controlled trial..." (p. 4) 2) "David A. Raitzer (draitzer@adb.org) is a senior economist... and Milan Thomas (mthomas@adb.org) is a country economist... Asian Development Bank." (p. 2) 3) "The study also would not have been possible without the implementation efforts of ADB colleagues... The pilot activities were supported... financed... through the Asian Development Bank" (p. 4) Detailed Analysis: The criterion requires the study to be conducted independently from the intervention designers/implementers. Here, the authors include staff from the Asian Development Bank (ADB), which funded, supported ("implementation efforts of ADB colleagues"), and helped conduct the trial ("Together with DepEd, we conducted..."). The authors are evaluating a project implemented by their own organization. There is no statement of an independent external evaluation team conducting the analysis. Criterion I is not met because the authors are affiliated with the organization that designed and implemented the intervention.
  • Y

    Year Duration

    • The intervention and data collection spanned two full academic years, exceeding the one-year requirement.
    • The endline survey was administered in May 2024, marking the completion of the second academic year during which students had access to the tablets.
    • Relevant Quotes: 1) "In August 2022, all students in the treatment schools received a tablet..." (p. 10) 2) "The endline survey was administered in May 2024, marking the completion of the second academic year during which students had access to the tablets." (p. 12) Detailed Analysis: The study tracks the intervention and outcomes over a period spanning two academic years (2022-2023 and 2023-2024). This satisfies the requirement for the study to last at least one full academic year. Criterion Y is met because the tracking period covered two full academic years.
  • B

    Balanced Control Group

    • The study explicitly tests the provision of hardware and software resources (tablets and modules) as the primary treatment, justifying the resource difference with the control group.
    • Provision of tablets and digitized learning modules (DLMs)... will increase student performance... relative to a baseline context of instruction via paper-based modules.
    • Relevant Quotes: 1) "The intervention provided selected junior high school students with low-cost tablets, complemented by installing antennae to establish local area networks..." (p. 8) 2) "Hypothesis 1) Provision of tablets and digitized learning modules... will increase student performance... relative to a baseline context of instruction via paper-based modules." (p. 9) 3) "To supplement the prevailing remote learning environment, students in treated schools were provided tablets..." (p. 8) Detailed Analysis: The intervention group received additional resources (tablets, LAN, solar panels) compared to the control group which relied on paper-based modules. Usually, this would be a failure of the Balanced Resources criterion. However, the standard allows for an exception if the intervention "explicitly tests the impact of additional resources... as the primary treatment variable." Here, the explicit hypothesis is testing the "Provision of tablets and digitized learning modules" against the baseline instruction. The provision of the hardware/technology *is* the treatment being evaluated. Therefore, the control group receiving the standard "business as usual" without these extra resources is consistent with the study's design intent. Criterion B is met because the study explicitly tests the provision of additional technological resources as the primary treatment variable.

• Level 3 Criteria

  • R

    Reproduced

    • This is an original study and no independent replication of this specific intervention is reported.
    • Relevant Quotes: 1) "The study presented in this paper tests whether educational technology in the form of computer-assisted instruction (CAI) could help..." (p. 6) 2) "This study illustrates both the promise and perils of using CAI..." (p. 25) Detailed Analysis: The "Learning at the Last Mile" study evaluates a specific pilot project in the Philippines. A search for independent replications of this specific tablet intervention in Philippine "Last Mile Schools" by other research teams yielded no results in peer-reviewed journals. The references cited within the paper are related studies, but not replications of this specific protocol. Criterion R is not met because no independent replication of this specific trial is reported.
  • A

    All-subject Exams

    • The study measured only Math and English outcomes, excluding Science which was part of the intervention content, and failed the standardized exam prerequisite.
    • The evaluation focuses on monitoring student outcomes in math and English assessments
    • Relevant Quotes: 1) "The evaluation focuses on monitoring student outcomes in math and English assessments..." (p. 8) 2) "A total of 249 DLMs were developed... across English, math and science..." (p. 10) Detailed Analysis: The ERCT standard for 'All-subject Exams' requires measuring impact on all main subjects. While the intervention materials (DLMs) included Science ("English, math and science"), the study only assessed outcomes in Math and English. Science was not measured. Furthermore, this criterion requires that the assessments be standardized (Criterion E met). Since Criterion E was not met (custom tests), Criterion A is automatically not met. Criterion A is not met because only two subjects were tested despite materials covering three, and standard exams were not used.
  • G

    Graduation Tracking

    • The study stopped tracking students once they graduated and did not collect or report graduation data.
    • endline data collection by DepEd was restricted to students who were in Grades 7 and 8 at baseline... [students] in Grades 9 and 10 at baseline had graduated
    • Relevant Quotes: 1) "By 2024, students who were in Grades 9 and 10 at baseline had graduated from junior high school, and thus, endline data collection by DepEd was restricted to students who were in Grades 7 and 8 at baseline..." (p. 8) 2) "This report focuses on that subset of students and learning progression between early 2022 and mid-2024." (p. 8) Detailed Analysis: The criterion requires tracking participants until their graduation. The study explicitly excluded the older cohorts (Grade 9 and 10 at baseline) from the endline specifically *because* they had graduated. For the younger cohorts (Grade 7 and 8 at baseline), the study tracked them until Grades 9 and 10. A search for follow-up papers tracking the younger cohort to graduation yielded no results, which is expected given the recent publication date (Dec 2025). Criterion G is not met because the study ceased data collection for students who graduated and did not report graduation outcomes.
  • P

    Pre-Registered

    • The paper does not provide any reference to a pre-registered study protocol or registry ID.
    • Relevant Quotes: 1) "We identified 42 eligible schools for the tablet intervention pilot using a list..." (p. 10) 2) "Together with the Department of Education, we conducted a 42-school randomized controlled trial..." (p. 4) Detailed Analysis: A search of the AEA RCT Registry and other standard databases for pre-registration of the "Learning at the Last Mile" Philippines study by Glewwe et al. or ADB did not yield a specific registry entry or protocol published prior to the intervention start (2022). The paper itself does not cite a pre-registration number (e.g., AEARCTR-XXXX). Criterion P is not met because there is no evidence or citation of a pre-registered study protocol.