The Impact of an Instructional Intervention on the Science and Language Learning of Middle Grade English Language Learners

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation occurred at the section (classroom) level within each teacher, satisfying the class-level RCT requirement.
    • "For each teacher, 2 sections were randomly assigned to the intervention, Project QuEST, and 2 sections were randomly assigned to the district curriculum."
    • Relevant Quotes: 1) "For each teacher, 2 sections were randomly assigned to the intervention, Project QuEST, and 2 sections were randomly assigned to the district curriculum." (Abstract, p. 345) 2) "The research involved 40 sections of students receiving their science instruction from one of 10 teachers at 5 middle schools." (p. 351) 3) "Each teacher contributed 4 sections, 2 of which were randomly assigned to treatment and 2 of which were assigned to typical instruction." (pp. 351-352) 4) "At the middle grades level, a section is a classroom of students. In this district, middle grades teachers were generally assigned six sections of students a day." (footnote 5, p. 351) Detailed Analysis: The paper explicitly defines a "section" as a classroom of students (footnote 5) and describes randomisation as occurring at the section (i.e., classroom) level within each teacher: two of each teacher's four sections were randomly assigned to QuEST and two to the district curriculum. This is a clear class-level randomised design, with entire classrooms (not individual students within a single classroom) assigned to treatment or control. Criterion C is met because randomisation was conducted at the section (classroom) level, as required for a class-level RCT.
  • E

    Exam-based Assessment

    • The primary science and vocabulary outcome measures were custom, researcher-developed instruments rather than standardised exams.
    • "In addition, students were assessed in English for science and vocabulary knowledge using researcher-developed assessments aligned with the district's sixth-grade science curriculum..."
    • Relevant Quotes: 1) "In addition, students were assessed in English for science and vocabulary knowledge using researcher-developed assessments aligned with the district's sixth-grade science curriculum covered during the 9 weeks the intervention was in place..." (p. 355) 2) "The items for the science tests were selected from items released from past state science tests as well as from an item bank available from the publisher of the textbook both treatment and control classrooms were using." (p. 355) 3) "The researcher developed vocabulary measure consisted of both the general academic and discipline specific words that appeared in the textbook." (p. 355) 4) "All students in the 40 science sections were assessed using the vocabulary and passage comprehension subtests of the GRADE assessment, Form A, Level M, prior to the onset of treatment as a safeguard against possible unhappy randomization." (p. 355) Detailed Analysis: The study's primary outcome measures—the science knowledge test and the vocabulary test used at pretest and posttest—were both explicitly "researcher-developed," created by the study team and aligned to the specific curriculum content taught during the 9-week intervention window, not a widely-recognised standardised exam. Although some science items were drawn from released state test items and a publisher item bank, the overall instrument was assembled and validated by the research team for this study, rather than administered as an existing standardised test. The one genuinely standardised instrument used, the GRADE assessment, was administered only once, at pretest, as a check on the comparability of groups from randomisation—not as an outcome measure of the intervention's effect. Criterion E is not met because the primary outcome assessments (science and vocabulary) were researcher-developed rather than standardised exams.
  • T

    Term Duration

    • The intervention and outcome measurement together spanned only about 9 weeks, shorter than the one-term minimum required.
    • "Teachers taught two of their science sections using the QuEST materials and strategies for a period of 9 weeks and taught two other sections of science using the district standard curriculum over the same period."
    • Relevant Quotes: 1) "Teachers taught two of their science sections using the QuEST materials and strategies for a period of 9 weeks and taught two other sections of science using the district standard curriculum over the same period." (p. 355) 2) "The instructional materials consisted of one 5-week unit and one 4-week unit; both focused on Living Systems and the Environment." (p. 352) 3) "The pretest for Unit 1 was collected prior to the onset of instruction for Unit 1, whereas the pretest for Unit 2 was collected at the end of instruction for Unit 1 along with the posttest for Unit 1 and prior to the onset of instruction for Unit 2." (pp. 355-356) Detailed Analysis: The intervention and the parallel control instruction both ran for a single continuous period of 9 weeks (a 5-week unit followed by a 4-week unit), with the posttest collected at the end of Unit 2, i.e. immediately at the end of the 9-week instructional period. Nine weeks (about two months) is well short of a full academic term, which the standard defines as approximately 3-4 months. There is no indication of any measurement occurring later than the end of the intervention itself. Criterion T is not met because the interval from intervention start to outcome measurement was only approximately 9 weeks, shorter than one academic term.
  • D

    Documented Control Group

    • The control group's curriculum, sample sizes, and baseline performance are clearly documented in the text and in Tables 1 and 3.
    • "In the district, the science curriculum was aligned with state and district standards, and at the sixth-grade level consisted of the Prentice Hall textbook and workbook, and district developed labs aligned with textbook content."
    • Relevant Quotes: 1) "In the district, the science curriculum was aligned with state and district standards, and at the sixth-grade level consisted of the Prentice Hall textbook and workbook, and district developed labs aligned with textbook content." (p. 352) 2) "Participants in this study included 890 students; 562 were ELLs and 328 were English proficient. Students' designation was based on district language proficiency testing. ELLs per class ranged from 3 to 31 (16-92% per class)." (p. 351) 3) Table 1 (p. 352) reports pretest and posttest sample sizes for Control Sections A and B, separately for each of the 10 teachers. 4) Table 3 (p. 358) reports pretest and posttest means, standard deviations, and Ns for both Vocabulary and Science, broken out by Treatment/Control group and by language status (English proficient vs. ELL). Detailed Analysis: The control condition is clearly documented: control sections continued with the district's standard, textbook-based sixth-grade science curriculum (Prentice Hall textbook/workbook plus district labs), described explicitly and contrasted with the QuEST materials. Sample sizes for control sections are broken down by teacher in Table 1, and baseline (pretest) performance for the control group is reported separately by language-proficiency subgroup in Table 3, alongside overall demographic information about the participant sample (890 students; 562 ELLs and 328 English proficient; district-level poverty and ethnicity statistics). Criterion D is met because the control group's instructional condition, sample sizes, and baseline performance are documented in sufficient detail for proper comparison.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was at the section (classroom) level within teachers and schools, not at the school level.
    • "The research involved 40 sections of students receiving their science instruction from one of 10 teachers at 5 middle schools."
    • Relevant Quotes: 1) "For each teacher, 2 sections were randomly assigned to the intervention, Project QuEST, and 2 sections were randomly assigned to the district curriculum." (Abstract, p. 345) 2) "The research involved 40 sections of students receiving their science instruction from one of 10 teachers at 5 middle schools." (p. 351) 3) "There are 10 middle schools in the district; five principals agreed to participate in the study." (p. 351) Detailed Analysis: Randomisation occurred at the section (classroom) level, nested within teacher, nested within school: every participating school and every participating teacher contributed both treatment and control sections simultaneously. No whole schools were randomly assigned to a single condition; instead, each of the 5 schools, and each of the 10 teachers within them, delivered both QuEST and the district curriculum to different sections in parallel. Criterion S is not met because randomisation was conducted at the classroom (section) level, not at the level of whole schools.
  • I

    Independent Conduct

    • The intervention's designers also delivered professional development, monitored fidelity, and collaborated on data analysis, with no independent evaluator identified.
    • "Professional development was provided by the Principal Investigator and science specialists hired by the project the help craft the curriculum."
    • Relevant Quotes: 1) "The materials development team consisted of science educators, as well as reading and ESOL specialists." (p. 353) 2) "Professional development was provided by the Principal Investigator and science specialists hired by the project the help craft the curriculum." (p. 354) 3) "A mentor who had been trained by the Principal Investigator and who had attended the professional development sessions worked with teachers on a weekly basis to help ensure that the curriculum was implemented with fidelity." (p. 354) 4) "Address correspondence to Diane August, Center for Applied Linguistics..." (p. 345); Diane August is the lead/corresponding author and is referred to as the study's Principal Investigator. Detailed Analysis: The same research team that developed Project QuEST (a materials development team including the Principal Investigator) also designed and delivered the teacher professional development, trained and supervised the weekly mentor who monitored implementation fidelity, and (through the co-authors at the Texas Institute for Measurement, Evaluation, and Statistics) analysed the resulting data. There is no statement in the paper of an independent, third-party organisation or evaluator conducting data collection or analysis separate from the intervention's designers; the mentor who assessed fidelity was trained and supervised by the same Principal Investigator who designed the intervention. Criterion I is not met because the intervention designers were also responsible for professional development, implementation monitoring, and (via co-author collaborators) the study's data analysis, with no independent evaluation team identified.
  • Y

    Year Duration

    • The study duration was only about 9 weeks, far short of a full academic year, and criterion T was also not met.
    • "Teachers taught two of their science sections using the QuEST materials and strategies for a period of 9 weeks..."
    • Relevant Quotes: 1) "Teachers taught two of their science sections using the QuEST materials and strategies for a period of 9 weeks and taught two other sections of science using the district standard curriculum over the same period." (p. 355) 2) "The instructional materials consisted of one 5-week unit and one 4-week unit; both focused on Living Systems and the Environment." (p. 352) Detailed Analysis: As established under criterion T, the entire intervention and its outcome measurement span only about 9 weeks, far short of the 75% of an academic year (roughly 7 months) required by criterion Y. Per the ERCT standard, since criterion T (Term Duration) is not met, criterion Y (Year Duration) is automatically not met. Criterion Y is not met because the tracked period (9 weeks) is far shorter than the required academic-year threshold, and criterion T is also not met.
  • B

    Balanced Control Group

    • The additional materials and professional development are integral, defining components of the QuEST treatment being tested against business-as-usual district instruction, and no extra class time was added.
    • "The intervention, called Project QuEST, was composed of two components that were not present in the district at the time of the intervention: instructional materials and professional development to help teachers use the instructional materials."
    • Relevant Quotes: 1) "The intervention, called Project QuEST, was composed of two components that were not present in the district at the time of the intervention: instructional materials and professional development to help teachers use the instructional materials." (p. 352) 2) "The instructional materials included a teacher guide and instructional charts, a student guide and instructional charts, and supplies for hands-on science activities. Professional development included three training sessions and ongoing weekly mentoring." (p. 352) 3) "Each week consisted of five 40-min lessons (the length of time allocated by the district for daily science instruction)." (pp. 352-353) 4) "The instructional materials were aligned with the Texas state and district sixth-grade science standards, covered the same content as was taught in the district's sixth-grade classrooms over the course of 12 weeks, and used the same textbook that was used in the regular classrooms." (p. 353) Detailed Analysis: Both treatment and control sections received the same amount of scheduled classroom time—five 40-minute lessons per week, the district's standard allocation for science—and covered the same content area (Living Systems and the Environment) using the same underlying textbook. No additional class time or budget was given to treatment sections; instead, the difference between conditions was the instructional materials (teacher and student guides, hands-on supplies) and professional development (training sessions, weekly mentoring) provided to teachers to help them teach the same content differently. The paper explicitly frames these two components—materials and professional development—as the two things that define "Project QuEST" and that were "not present in the district" for the control condition. Applying the criterion B decision tree: extra resources are present (materials and PD not previously available), and these resources are explicitly framed as the treatment variable itself (the two defining components of "Project QuEST" being tested against business-as-usual district instruction), so the control group may legitimately remain at the standard business-as-usual level without matching materials or PD. No extra instructional time was given to the treatment group. Criterion B is met because the additional materials and professional development are integral components of the QuEST treatment being tested against the district's standard "business as usual" curriculum, and no additional class time was given to the treatment group.

• Level 3 Criteria

  • R

    Reproduced

    • The paper presents itself as the first study of its kind; an internet-backed search found no independent replication by a different research team, only a 2025 follow-on study sharing most of the same authors.
    • "...to our knowledge this study represents the first such demonstration in a randomized controlled experiment that gains in content area knowledge in Science and gains in Vocabulary are possible for both ELL and English proficient students..."
    • Relevant Quotes: 1) "Although much work remains to be done, to our knowledge this study represents the first such demonstration in a randomized controlled experiment that gains in content area knowledge in Science and gains in Vocabulary are possible for both ELL and English proficient students using a common approach to instruction..." (pp. 371-372) 2) "Of all research published between 1982 and 2009 in English in settings where English is the main medium of science instruction in elementary and secondary schools this is only published experimental study we know of that has found significant intervention effects for both ELLs and English proficient students in science knowledge and science vocabulary." (p. 372) 3) From internet-backed search: August, Branum-Martin, Duguay, Powell, Carlson, Francis, & Echevarria (2025), "The impact of an instructional intervention on the science knowledge and academic language of middle-grade English as additional language (EAL) learners and their English proficient classmates," Journal of Research on Educational Effectiveness, DOI: 10.1080/09500693.2024.2419985. Abstract: "This waitlist control study evaluated whether an intervention designed to boost science knowledge and academic language benefited both EAL learners and their English proficient peers... Across nine schools over two years, researchers employed a randomized controlled design in year one, then implemented the intervention school-wide in year two." Detailed Analysis: The paper itself states this 2009 study was the first, and at the time the only, published experimental demonstration of its kind, and it references no prior or contemporaneous independent replication. A literature search of the 53 papers citing this study (via Semantic Scholar) found no independent replication of this specific Project QuEST randomized controlled trial conducted by a different research team. The closest related follow-on work identified is the 2025 study above, which tests a related inquiry-based science/academic-language intervention across nine schools; however, its author team substantially overlaps with the original study (August, Branum-Martin, and Francis are co-authors of both papers), so it does not constitute an independent replication by a different research team as required by criterion R. No other peer-reviewed replication of this specific study by an unrelated research group was located. Criterion R is not met because no independent replication of this specific study by a different research team was found, either within the paper or through internet-backed search of the citing literature.
  • A

    All-subject Exams

    • Criterion E was not met, and only science and vocabulary were assessed, not all main subjects, so criterion A is not met.
    • Relevant Quotes: 1) "In addition, students were assessed in English for science and vocabulary knowledge using researcher-developed assessments..." (p. 355) 2) "Treatment effects were tested separately for science knowledge and vocabulary using analysis of covariance (ANCOVA)..." (p. 358) Detailed Analysis: Per the ERCT standard's instructions for criterion A, since criterion E (Exam-based Assessment) is not met, criterion A is automatically not met. Independently, the study only measured science content knowledge and English vocabulary; no other core subjects (e.g., mathematics, reading comprehension, social studies) were assessed as outcomes, and no exception rationale for a specialised, single-subject focus is provided. Criterion A is not met because criterion E was not met, and because only science and vocabulary outcomes were assessed, not all main subjects.
  • G

    Graduation Tracking

    • Criterion Y was not met, and no publication tracking this study's cohort of students through graduation was found in the paper or via internet-backed search.
    • Relevant Quotes: 1) "Teachers taught two of their science sections using the QuEST materials and strategies for a period of 9 weeks..." (p. 355) 2) No statement anywhere in the Results, Discussion, Limitations, or Conclusion sections mentions any follow-up data collection beyond the immediate posttest at the end of the 9-week intervention. 3) "These shortcomings were addressed in a follow-up study in the same district." (p. 369) — this follow-up is described only in the context of addressing teacher-training and implementation shortcomings, with no indication that it extended outcome tracking of the same student cohort. 4) From internet-backed search: no publication was found tracking the original 2007-2008 cohort of 890 sixth-grade students from this study through to graduation. The related 2025 study by August, Branum-Martin, Duguay, Powell, Carlson, Francis, and Echevarria ("The impact of an instructional intervention on the science knowledge and academic language of middle-grade English as additional language (EAL) learners...", Journal of Research on Educational Effectiveness) is a separate randomized waitlist-control trial conducted across nine different schools over two years, not a longitudinal follow-up of this study's original student cohort. Detailed Analysis: Per the ERCT standard's instructions for criterion G, since criterion Y (Year Duration) is not met, criterion G is automatically not met. Independently, outcome data collection in this paper stopped at the immediate posttest following the 9-week intervention period. The paper mentions a "follow-up study in the same district" (p. 369), but this is described only as addressing implementation and professional-development shortcomings, not as tracking the same students' outcomes. An internet-backed search of the citing literature did not locate any publication tracking this study's cohort of students to graduation. Criterion G is not met because criterion Y was not met, and because no evidence of graduation tracking of this study's cohort was found either in the paper or through internet-backed search.
  • P

    Pre-Registered

    • No pre-registration statement or registry reference is found anywhere in the paper, and none was located via internet-backed search.
    • Relevant Quotes: No quotes referencing a study registry, registration ID, or pre-registration date were found anywhere in the paper, including the Method, Research Procedures, or Acknowledgments sections. Detailed Analysis: The paper contains no mention of the study protocol, hypotheses, or analysis plan having been publicly registered on any registry platform (e.g., ClinicalTrials.gov, ISRCTN, AEA RCT Registry) prior to data collection. The Acknowledgments section only credits the funding source (Institute of Education Sciences Grant R305A050056) and research assistants, with no reference to pre-registration. An internet-backed search for a pre-registration record associated with this study found none; this absence is consistent with the fact that this study's data collection occurred in the 2006-2007 school year, predating widespread adoption of pre-registration in education research (the AEA RCT Registry, a primary registry for this field, was not established until 2013). Criterion P is not met because no pre-registration statement, registry link, or registration date is provided anywhere in the paper, and no pre-registration record was found via internet-backed search.