Computer assisted English language learning in Costa Rican elementary schools: an experimental study

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the whole-school level, with each school having only a single third-grade classroom, so school-level assignment automatically satisfies the weaker class-level requirement.
    • "A power analysis was used to determine the required number of schools and students for this project ... a randomized block design (with each school as a unit of block) was implemented to select students randomly to meet the number needed and to ensure that each school participated in the project."
    • Relevant Quotes: 1) "A power analysis was used to determine the required number of schools and students for this project through Optimal Design ... With the effect size predetermined at .30 (small), alpha of .05, and intraclass correlation (ICC) of .05, a statistical power of .98 corresponds to a sample size of 80 schools (with an average of 15 students per school). In cases where there were more than 15 students on one campus, a randomized block design (with each school as a unit of block) was implemented to select students randomly to meet the number needed and to ensure that each school participated in the project." (p. 107) 2) "Note that we did not conduct a stratified randomization by school type, i.e., rural, urban, and urban/marginalized; instead, the design was a complete randomization based on the list of schools provided by the Costa Rica Multilingual Department from the previous school year for an estimation of the number of students who would attend each school." (p. 107) 3) "This project started with 868 third graders from 77 primary schools randomly selected and assigned within the Province of Alajuela, Costa Rica." (p. 107) 4) Table 1, "Schools and student participants by condition at the initiation and end of the project," lists whole numbers of schools per condition (Control 24, Treatment A 26, Treatment B 27 at initiation), confirming assignment occurred at the school unit rather than within-school. (p. 108) 5) "The classroom/teacher level was not included as a higher level, because there was one classroom per school, and adding both classroom and school levels to the model could have resulted in computational difficulty..." (p. 112) Detailed Analysis: The authors explicitly describe assigning entire schools to one of the three conditions (Control, Treatment A, Treatment B), drawing from a province-wide list of eligible schools and randomly allocating them whole. Table 1 confirms this by reporting counts of schools, not individual students within a shared classroom, per condition. Furthermore, the paper clarifies that each participating school had only one third-grade classroom, meaning that a school-level randomisation is, in this specific design, identical to a class-level randomisation with no risk of within- classroom or within-school contamination between conditions. Under the ERCT Standard, a stronger school-level RCT automatically satisfies the weaker class-level RCT requirement. Criterion C is met because randomisation occurred at the whole-school (and thus whole-class) level, which is stronger than the minimum class-level requirement.
  • E

    Exam-based Assessment

    • The study measured oral English proficiency using the WMLS-R, a norm-referenced, standardized instrument with documented reliability and validity, not a custom-built test.
    • "WMLS-R (Woodcock, Munoz-Sandoval, Ruef, & Alvarado, 2005) is a norm-referenced, standardized instrument to measure language proficiency in reading, writing, listening, and comprehension."
    • Relevant Quotes: 1) "WMLS-R (Woodcock, Munoz-Sandoval, Ruef, & Alvarado, 2005) is a norm-referenced, standardized instrument to measure language proficiency in reading, writing, listening, and comprehension. In the current project, a total of four subtests from WLMS-R were given to monitor participants' oral English development, including Picture Vocabulary, Verbal Analogies, Understanding Directions, and Story Recall." (p. 111) 2) "According to the authors, the split-half reliability ranges from .76 to .97 for the test, and content validity was well established with concurrent validity ranging from .70s to .80s with other standardized tests that measure oral language (IDEA Proficiency Test, Language Assessment Scale), intelligence (Wechsler Adult Intelligence Scale), and academic achievement (Wide Range Achievement Test and Woodcock Johnson III Tests of Achievement)..." (p. 111) 3) "W-scores were used for analysis to reflect the developmental pattern. According to the technical manual of WMLS-R, the W-score is a special transformation of the Rasch scale, which has equal-interval that is well suited for use as growth scale..." (p. 111-112) Detailed Analysis: The WMLS-R is an established, norm-referenced, standardized instrument published by an independent test developer (Woodcock, Munoz-Sandoval, Ruef, & Alvarado, 2005), with reported psychometric properties (reliability .76-.97, concurrent validity .70s-.80s against other recognized instruments). It was not designed by the study authors for the purposes of this specific project, and four of its standard subtests were applied unmodified to measure oral English proficiency. Criterion E is met because the outcome measure is a widely recognized, independently developed standardized assessment instrument (WMLS-R), not a custom test built for this study.
  • T

    Term Duration

    • The intervention and post-test spanned a full 25-week period, which exceeds the minimum one-term (roughly 3-4 month) requirement.
    • "The intervention of Treatment A included technology-based instruction every day, or five days per week, for 25 weeks."
    • Relevant Quotes: 1) "This study presents first-year findings of a 25-week longitudinal project derived from a two-year longitudinal randomized trial study..." (p. 103, Abstract) 2) "The intervention of Treatment A included technology-based instruction every day, or five days per week, for 25 weeks." (p. 108) 3) "The Control classrooms included teacher-led instruction every day, or five days per week, for 25 weeks..." (p. 111) 4) "As a result, there were 816 students from 76 schools remaining at the completion of the project for the first year ending November 2010, with 703 students having both pre-test and post-test data." (p. 107-108) Detailed Analysis: The intervention ran continuously for 25 weeks (roughly 5.5-6 months) across all three conditions, with the post-test administered at the conclusion of that period, near the end of the school year (November 2010). A 25-week interval clearly exceeds the ERCT minimum requirement of at least one academic term (approximately 3-4 months) between intervention start and outcome measurement. Criterion T is met because outcomes were measured roughly 25 weeks after the intervention began, well beyond the one-term minimum.
  • D

    Documented Control Group

    • The Control/Comparison group's curriculum, instruction time, and school/student demographics are described in detail and compared against the treatment conditions.
    • "The Control group, consisting of typical English instruction with a teacher only, is based on the English language development curriculum guidelines of the Costa Rican MEP."
    • Relevant Quotes: 1) "The Control group, consisting of typical English instruction with a teacher only, is based on the English language development curriculum guidelines of the Costa Rican MEP. According to the MEP (Relanzamiento de la Educacion Costarricense, 2010), the teaching of English in Costa Rica is performed in cycles. Cycle 1 was implemented during our study." (p. 109-110) 2) "The Control classrooms included teacher-led instruction every day, or five days per week, for 25 weeks; however, based on observations by our on-site research fidelity coordinator, there were three days, 150 minutes on average per week." (p. 111) 3) "Table 6. School demographics by condition in references to the population in Alajuela Province," reporting number of schools, average students per school, percent rural, teachers per school, and student-to-teacher ratio broken out for the Control condition alongside Treatment A and B. (p. 119) 4) Table 3 reports Control group pre-test and post-test means and standard deviations for all four WMLS-R subtests. (p. 115) Detailed Analysis: The paper devotes an entire subsection to describing the Control condition's curriculum content (listening and speaking objectives under MEP Cycle 1), its actual observed instructional time (150 minutes/week across three days, based on fidelity observation), and provides detailed baseline demographic and academic comparisons (Tables 3 and 6) between the Control group and the two treatment groups, including school type, rural/urban composition, teacher ratios, and pre-test scores. Criterion D is met because the Control group's curriculum, instructional time, and demographic/ baseline characteristics are clearly and quantitatively documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Entire schools, not individual students or single classes within a shared school, were the unit of random assignment to Control, Treatment A, or Treatment B.
    • "This project started with 868 third graders from 77 primary schools randomly selected and assigned within the Province of Alajuela, Costa Rica."
    • Relevant Quotes: 1) "This project started with 868 third graders from 77 primary schools randomly selected and assigned within the Province of Alajuela, Costa Rica." (p. 107) 2) "In cases where there were more than 15 students on one campus, a randomized block design (with each school as a unit of block) was implemented to select students randomly to meet the number needed and to ensure that each school participated in the project." (p. 107) 3) Table 1 reports 24 schools assigned to Control, 26 to Treatment A, and 27 to Treatment B at initiation (77 schools total), i.e., whole schools per condition. (p. 108) 4) "The classroom/teacher level was not included as a higher level, because there was one classroom per school..." (p. 112) Detailed Analysis: "School" here refers to the educational institution (an elementary school in Alajuela Province), which is exactly the unit ERCT defines for the S criterion. The paper's sampling frame is a province-wide list of schools, and randomisation (with power-analysis-derived sample size of ~80 schools) was performed at that level, not by sampling individual students from a shared pool of classes. Each school, once assigned, delivered only one condition to its entire third-grade cohort. Criterion S is met because whole schools were the unit of random assignment to the three study conditions.
  • I

    Independent Conduct

    • No statement documents that the study was conducted by an evaluator independent of the intervention designers; staff from the implementing Costa Rica Multilingue Foundation are themselves co-authors.
    • "Katherine Stanley is the Academic Director of the Costa Rica Multilingue Foundation and served as the lead field coordinator of this research project."
    • Relevant Quotes: 1) "Marta Blanco ... currently serves as the Executive Director of the Costa Rica Multilingue Foundation." (p. 124, Notes on contributors) 2) "Katherine Stanley is the Academic Director of the Costa Rica Multilingue Foundation and served as the lead field coordinator of this research project." (p. 124, Notes on contributors) 3) "The significance of our study, funded by the Inter-American Development Bank and the Costa Rica United States Foundation..." (p. 106-107) 4) "According to the software company for Treatment B, it has been determined that the Treatment B software has had a positive impact on students' literacy and language ability as measured by Illinois Snapshot of Early Literacy (ISEL) and California English Language Development Test (CELDT)." (p. 109) Detailed Analysis: While the underlying CALL software for Treatment A and Treatment B appears to originate from external technology vendors ("the software company for Treatment B"), the paper contains no explicit statement that the trial's field implementation, data collection, or analysis were carried out by an evaluator independent of the intervention's designers/implementers. To the contrary, the Notes on Contributors confirm that co-author Marta Blanco was Executive Director, and co-author Katherine Stanley was Academic Director and "lead field coordinator," of the Costa Rica Multilingue Foundation, the organization that ran Project EILE and selected/implemented the intervention in schools. These same individuals are co-authors of the paper reporting the results, and no acknowledgement, disclosure, or methods statement describes a third-party or externally contracted evaluation team responsible for data collection or analysis, distinct from the implementing Foundation staff. Criterion I is not met because the paper provides no quoted evidence that the evaluation was conducted independently of the team implementing the intervention; instead, the implementing Foundation's own staff are co-authors of the study.
  • Y

    Year Duration

    • The tracked intervention and follow-up period was only 25 weeks, well under the 75%-of-an-academic-year threshold required by the Year Duration criterion.
    • "This study presents first-year findings of a 25-week longitudinal project derived from a two-year longitudinal randomized trial study at the elementary school level in Costa Rica..."
    • Relevant Quotes: 1) "This study presents first-year findings of a 25-week longitudinal project derived from a two-year longitudinal randomized trial study at the elementary school level in Costa Rica on effective computer- assisted language learning (CALL) approaches..." (p. 103, Abstract) 2) "The intervention of Treatment A included technology-based instruction every day, or five days per week, for 25 weeks." (p. 108) 3) "As a result, there were 816 students from 76 schools remaining at the completion of the project for the first year ending November 2010..." (p. 107-108) Detailed Analysis: The reported intervention and outcome measurement window is 25 weeks, roughly 5.5-6 months. A typical academic year of ~9-10 months (about 39-40 weeks) means 75% coverage would require approximately 29-30 weeks at minimum. Twenty-five weeks falls short of that 75% threshold, and the authors themselves frame this paper as only the "first-year findings" of a still-larger two-year project, implying that a full-year analysis was not yet available at the time of this report. Criterion Y is not met because the tracked period (25 weeks) does not reach 75% of a full academic year.
  • B

    Balanced Control Group

    • The Control group's observed instructional time (150 minutes/week) met or exceeded that of both Treatment A (67 minutes/week) and Treatment B (127 minutes/week), so no imbalance favoring the intervention groups is present; the CALL technology is also the explicit treatment variable being tested.
    • "The Control classrooms included teacher-led instruction every day, or five days per week, for 25 weeks; however, based on observations by our on-site research fidelity coordinator, there were three days, 150 minutes on average per week."
    • Relevant Quotes: 1) "The average time-on-task for Treatment A was 67 minutes per week." (p. 108) 2) "The average time-on-task for Treatment B was 127 minutes per week." (p. 109) 3) "The Control classrooms included teacher-led instruction every day, or five days per week, for 25 weeks; however, based on observations by our on-site research fidelity coordinator, there were three days, 150 minutes on average per week." (p. 111) 4) "There is no variation regarding time-on-task within the Control/Comparison condition, because it was not possible to document the accurate number of minutes each student spent learning English in class. (This constitutes a limitation within the study.)" (p. 121) Detailed Analysis: Applying the criterion B decision procedure: extra resources (laptops, headsets, and CALL software) are present in the intervention groups relative to the Control's teacher-only instruction, but the paper explicitly frames "effective computer-assisted language learning (CALL) approaches" as the primary treatment variable being tested against typical teacher-led "business as usual" instruction, so this resource is INTEGRAL to the design rather than a separable add-on (spec 5c). Even setting the treatment-variable framing aside, the actual measured instructional time given to the Control group (150 minutes/week, per independent fidelity observation) was not less than, and in fact exceeded, the time-on-task logged for both Treatment A (67 minutes/week) and Treatment B (127 minutes/week), so the CONTROL_MATCHES_RESOURCES branch is also satisfied. Under both branches of the decision tree the criterion resolves to met. Criterion B is met because the additional CALL technology is the explicit treatment variable under study, and independently the Control group's measured instructional time was comparable to (and not less than) that of the treatment groups.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific Project EILE trial by a different research team was found in the paper or via external search of citing literature.
    • Relevant Quotes: 1) "Finally, we recommend that additional research be conducted with groups that move through kindergarten, first grade, and second grade longitudinally to determine cohort effects in learning English via CALL instruction in EFL countries." (p. 103, Abstract, and p. 122, Future research) 2) "We recommend that additional research be conducted to continue the study longitudinally. Also, in the second year of intervention, it is recommended to test teacher effect..." (p. 122) Detailed Analysis: The paper itself does not report any prior or concurrent independent replication of this specific Project EILE trial; the "Future research" section only recommends continuing the same project longitudinally and testing additional cohorts, not an independent replication by another team. An internet search was conducted via Crossref, Semantic Scholar, and OpenAlex to review the approximately 24-30 works citing this paper (spanning 2016-2026), including systematic reviews and meta-analyses of CALL and educational technology (e.g., Norman 2023; Silverman, Keane, Darling-Hammond, & Khanna 2024; Sijali, Poudel, & Dahal 2026), and none of these citing works report an independent replication of this specific Costa Rican Project EILE trial (same intervention, design, and population) by a different research team; they are literature reviews, meta-analyses, or unrelated CALL studies that merely cite this paper as background. Criterion R is not met because no independent replication of this specific study was found either in the paper or through a search of the literature citing it.
  • A

    All-subject Exams

    • The study measured only oral English language proficiency via WMLS-R subtests, with no assessment of other core school subjects such as mathematics or science.
    • "Four subtests from Woodcock Munoz Language Survey- Revised (WMLS-R), a norm-referenced, standardized instrument, were selected to monitor participants' oral English development."
    • Relevant Quotes: 1) "Four subtests from Woodcock Munoz Language Survey- Revised (WMLS-R), a norm-referenced, standardized instrument, were selected to monitor participants' oral English development." (p. 103, Abstract) 2) "In the current project, a total of four subtests from WLMS-R were given to monitor participants' oral English development, including Picture Vocabulary, Verbal Analogies, Understanding Directions, and Story Recall." (p. 111) 3) There is no mention anywhere in the Results, Discussion, or Instrument sections of assessments in mathematics, science, or other core subjects for these third-grade students. Detailed Analysis: Criterion E is met (WMLS-R is a standardized instrument), which is the prerequisite for A, but the study exclusively measured oral English proficiency (Picture Vocabulary, Verbal Analogies, Understanding Directions, Story Recall) and did not assess outcomes in other main subjects taught to Costa Rican third graders, such as mathematics, science, or Spanish language arts. The paper's stated exception pathway (specialized upper- secondary/vocational interventions) does not apply here, since this is a general elementary EFL intervention, not a specialized upper-secondary/vocational program, and no rationale is given for omitting other subjects. Criterion A is not met because only oral English proficiency was assessed, with no coverage of other core subjects and no applicable exception.
  • G

    Graduation Tracking

    • Criterion Y (Year Duration) is not met, and the paper reports only first-year, 25-week findings with no follow-up located through graduation.
    • "This study presents first-year findings of a 25-week longitudinal project derived from a two-year longitudinal randomized trial study at the elementary school level in Costa Rica..."
    • Relevant Quotes: 1) "This study presents first-year findings of a 25-week longitudinal project derived from a two-year longitudinal randomized trial study..." (p. 103, Abstract) 2) "We recommend that additional research be conducted to continue the study longitudinally. Also, in the second year of intervention, it is recommended to test teacher effect with teachers involved in the instruction as opposed to only CALL instruction as was done in the current year study reported herein." (p. 122, Future research) 3) "It is recommended that the continuation of this project be considered so that more data points become available to model students' academic oral language development in English..." (p. 122) Detailed Analysis: The reported study covers only the first year (25 weeks) of a planned two-year project involving third-grade students, and outcomes were measured at the end of that 25-week window rather than through graduation from elementary school. The authors' own "Future research" section frames continued tracking as a recommendation for subsequent work, confirming that no graduation-level follow-up occurred within this paper. An internet search via Crossref, Semantic Scholar, and OpenAlex for subsequent publications by the same author team (Alvarez-Marinelli, Blanco, Lara-Alecio, Irby, Tong, Stanley, Fan) reporting second-year or later Project EILE results, or any longer-term follow-up of this cohort toward graduation, did not locate any such paper. Since the weaker Year Duration criterion (Y) is also not met, Graduation Tracking cannot be met either. Criterion G is not met because tracking stopped at the end of the first 25-week year, far short of graduation, no follow-up publication tracking this cohort further was found via literature search, and the weaker Year Duration criterion was also not met.
  • P

    Pre-Registered

    • No statement anywhere in the paper references a pre-registration platform, registration ID, or registration date for the study protocol, and no such registration was located via external search.
    • Relevant Quotes: 1) A full review of the Abstract, Research method, Research questions, Data analysis plan, Acknowledgements, and Notes sections reveals no mention of a trial registry (e.g., ClinicalTrials.gov, ISRCTN, AEA RCT Registry) or any pre-registration date. 2) "The purpose of our study, a component part of a larger two-year Costa Rican random trial experimental study, is to determine to what extent do third-grade students differ from pre-test to post-test in English language level among the three randomly assigned groups..." (p. 106-107) -- states study purpose and design but does not reference any pre-registered protocol. Detailed Analysis: The ERCT Standard requires an explicit, quotable reference to a pre-registered protocol (hypotheses, methods, planned analyses) published before data collection began. No such reference, registry link, or registration date appears anywhere in this paper. An external search of the AEA Social Science Registry and general academic databases for a "Project EILE" or Costa Rica Multilingue Foundation CALL trial registration did not locate any pre-registration record, consistent with the absence of any such statement in the text (the study also predates widespread pre- registration norms in education RCTs, being conducted in 2010 and published in 2014). Criterion P is not met because no pre-registration statement or registry reference is present in the text, and none was found through external search.