Helping ELLs Meet the Common Core State Standards for Literacy in Science: The Impact of an Instructional Intervention Focused on Academic Language

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation occurred at the classroom (section) level within each teacher, satisfying the class-level RCT requirement.
    • "Each teacher chose four of their sections for the study, two of which were randomly assigned to the treatment condition and two of which were assigned to the control condition." (p. 59)
    • Relevant Quotes: 1) "Sixty sections taught by 15 teachers in seven middle schools were randomized within teacher to the intervention or the control condition." (Abstract, p. 54) 2) "Each teacher chose four of their sections for the study, two of which were randomly assigned to the treatment condition and two of which were assigned to the control condition." (p. 59) 3) "4At the middle grades level, a 'section' is a classroom of students. In this district, middle grades teachers were generally assigned six sections of students per day." (footnote, p. 59) Detailed Analysis: The paper explicitly states that randomisation was conducted "within teacher" at the level of the "section," which is defined by the authors themselves as "a classroom of students." Each participating teacher had four of their classroom sections randomly split, two to treatment and two to control. This is a textbook class-level (entire classroom) randomisation: whole sections/classes, not individual students within a single class, were assigned to condition. There is no indication that randomisation occurred at the within-classroom student level, so contamination of the type the C criterion is meant to guard against (individual students in the same class receiving different conditions) is avoided. Criterion C is met because the unit of randomisation was the classroom section, which satisfies the class-level RCT requirement.
  • E

    Exam-based Assessment

    • The study used the GRADE, a widely recognised norm-referenced standardised reading test, as a formal outcome measure alongside researcher-developed assessments.
    • "All students in the 60 science sections were assessed using the word meaning subtest of the Group Reading Assessment and Diagnostic Evaluation (GRADE) Level M, prior to and following the intervention. The GRADE is a norm-referenced diagnostic reading test..." (p. 63)
    • Relevant Quotes: 1) "All students in the 60 science sections were assessed using the word meaning subtest of the Group Reading Assessment and Diagnostic Evaluation (GRADE) Level M, prior to and following the intervention. The GRADE is a norm-referenced diagnostic reading test that assesses word reading, word meaning, and comprehension skills." (p. 63) 2) "In addition, students were assessed on academic language and science knowledge using researcher-developed assessments aligned with the district's sixth-grade science curriculum covered during the 15 weeks the intervention was in place." (p. 63) 3) "The items for the science tests were selected from items released from past state science tests as well as from an item bank available from the publisher of the textbook." (p. 63) Detailed Analysis: The two central outcome measures used to answer the study's main research questions—the curriculum-based measures (CBM) of academic language (vocabulary) and science knowledge—are explicitly researcher-developed instruments, tailored to the specific curriculum content taught during the intervention (though science items were partly drawn from released state test items and a publisher item bank, they were assembled into a custom instrument by the research team). Under a strict reading, this alone would not satisfy the standardised exam-based assessment criterion. However, the study additionally administered the GRADE, a widely used, commercially published, norm-referenced diagnostic reading test, as a formal pre/post outcome measure for all 1,309 students, with results formally analysed and reported (see Table 6, Table 7, and Table 8, "GRADE Vocabulary"). This is a genuine, externally validated standardised instrument, not created for this study, and its use is reported with reliability statistics (Table 2). Because the GRADE was a bona fide standardised exam-based assessment actually used to evaluate outcomes (even though results on it were non-significant), the criterion's core concern— that the study include a standard, widely recognised assessment rather than relying solely on a test invented to flatter the intervention— is satisfied. Criterion E is met because the study administered the GRADE, a widely recognised, norm-referenced standardised test, as one of its formal outcome measures, in addition to researcher-developed measures.
  • T

    Term Duration

    • Outcomes were measured after a roughly 15-week intervention period, which is consistent with one academic term.
    • "Over the course of the 15-week intervention, mentors observed and met with each teacher seven times." (p. 63)
    • Relevant Quotes: 1) "Over the course of the 15 units, teachers were expected to cover the same curricular content—Earth and Space Systems and Living Systems and Environment—in their treatment and control sections." (p. 60) 2) "Over the course of the 15-week intervention, mentors observed and met with each teacher seven times." (p. 63) 3) "The academic vocabulary and science tests were administered prior to and following instruction in each topic area (i.e., Earth and Space Systems, Living Systems and Environment)." (p. 65) 4) "All students in the 60 science sections were assessed using the word meaning subtest of the ... GRADE ... prior to and following the intervention." (p. 63) Detailed Analysis: The intervention consisted of 15 weekly units of five 40-minute lessons each, i.e. roughly 15 weeks, or approximately 3.5 months, of instruction. Primary outcome measures (GRADE, and the CBM academic-language and science tests) were administered before the intervention began and again after the full 15-week intervention concluded, so the interval from intervention start to final measurement spans the full 15-week period. Fifteen weeks is consistent with the ERCT Standard's definition of "a semester or equivalent (approximately 3-4 months)," so the measured interval from start to posttest meets the term-duration bar. Criterion T is met because outcomes were measured at the end of a roughly 15-week (about one academic term) intervention period, matching or exceeding the minimum term-duration requirement.
  • D

    Documented Control Group

    • The control group's size, demographic makeup, baseline equivalence, and standard instruction received are documented in detail.
    • "Treatment and control sections did not differ from each other with respect to composition for gender ... or number of ELL students ..." (p. 59)
    • Relevant Quotes: 1) "The control condition was the typical instruction carried out in science classrooms in the district." (p. 59) 2) "The number of students in each of the four sections for each teacher is presented in Table 1. The table indicates that sections ranged from a minimum of six to a maximum of 31 students per classroom." (p. 59) 3) "Treatment and control sections did not differ from each other with respect to composition for gender (M = 50% vs. 53% in treatment and control), t(58) = 1.18, p = .24, or number of ELL students (M = 35% vs. 36% in treatment and control), t(58) = 0.16, p = .87." (p. 59) 4) "There were no statistically significant differences at pretest between treatment and control groups (full tables of these preliminary analyses are available from the authors)." (p. 71) Detailed Analysis: The paper documents the control group in considerable detail: its size (Table 1, ranging 6-31 students per section, totals reported), its instructional content (typical district-standard instruction using the same textbook, workbook, and district labs as treatment, without the two extra intervention components), and formal statistical comparisons confirming the control group did not differ from the treatment group on gender composition, ELL composition, or pretest scores. This level of detail allows the reader to assess baseline comparability between arms and to see precisely what the "business as usual" control condition consisted of. Criterion D is met because the control group's size, composition, baseline equivalence, and the instruction it received are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred within teachers at the section level, not at the level of whole schools.
    • "Each teacher chose four of their sections for the study, two of which were randomly assigned to the treatment condition and two of which were assigned to the control condition." (p. 59)
    • Relevant Quotes: 1) "Sixty sections taught by 15 teachers in seven middle schools were randomized within teacher to the intervention or the control condition." (Abstract, p. 54) 2) "These students were in 60 sections in the seven middle schools where the principal elected to participate in the study." (p. 59) 3) "Each teacher chose four of their sections for the study, two of which were randomly assigned to the treatment condition and two of which were assigned to the control condition." (p. 59) Detailed Analysis: Randomisation in this study occurred strictly "within teacher," at the classroom-section level: every participating teacher taught both treatment and control sections simultaneously. Schools themselves were not randomised to condition—principals simply "elected to participate," and within each school, individual teachers (and their sections) were split between treatment and control. This means treatment and control conditions were both present within the same schools (and even administered by the same teachers), which is the opposite of the school-level design the S criterion requires. Criterion S is not met because randomisation was conducted at the classroom/section level within teachers, not at the level of whole schools.
  • I

    Independent Conduct

    • The intervention's designers directly trained and supervised all data collectors and observers; no independent third-party evaluator is described.
    • "Assessors and observers were research assistants who had been trained by the principal investigator." (p. 65)
    • Relevant Quotes: 1) "Assessors and observers were research assistants who had been trained by the principal investigator." (p. 65) 2) "The project principal investigator and project science specialists were responsible for training teachers." (p. 62) 3) "The person who provided mentoring was an experienced educator with a science background, who was also bilingual and local to the district in which the intervention occurred." (p. 62) 4) "There were a total of five observers. They had attended the professional development sessions and had been trained by the principal investigator." (p. 65) Detailed Analysis: The same research team that designed the QuEST 2 curriculum and professional development also trained the teachers, trained and supervised the mentor, trained the observers and assessors, and (per the Data Analysis section) fit and interpreted the statistical models themselves. There is no mention anywhere in the Methods of an external, third-party organisation independent of the developers conducting data collection or analysis; instead, all study personnel (mentor, observers, assessors) were trained directly by the project's principal investigator, who is also the intervention's designer and lead author. Criterion I is not met because the intervention designers (the research team led by the principal investigator) also directly trained and supervised all personnel who implemented, observed, and assessed the study, with no independent third-party evaluation described.
  • Y

    Year Duration

    • The study's intervention and outcome tracking spanned only about 15 weeks, far short of the 75%-of-a-year threshold required for Y.
    • "Over the course of the 15-week intervention, mentors observed and met with each teacher seven times." (p. 63)
    • Relevant Quotes: 1) "Over the course of the 15-week intervention, mentors observed and met with each teacher seven times." (p. 63) 2) "Over the course of the 15 units, teachers were expected to cover the same curricular content ... five 40-min weekly sessions." (p. 60) 3) "The academic vocabulary and science tests were administered prior to and following instruction in each topic area (i.e., Earth and Space Systems, Living Systems and Environment)." (p. 65) Detailed Analysis: The intervention and its associated outcome measurement span approximately 15 weeks, roughly 3.5 months. The ERCT Standard's Y criterion requires tracking of outcomes across at least 75% of a full academic year (roughly 9-10 months, i.e. approximately 27 weeks or more). Fifteen weeks represents only about 40% of a typical academic year, well short of the required 75% threshold, and there is no indication in the paper of any longer-term follow-up assessment beyond the end of the 15-week intervention. Criterion Y is not met because the study tracked outcomes for roughly 15 weeks, far short of 75% of an academic year.
  • B

    Balanced Control Group

    • Instructional time was equal across arms, and the extra materials/professional development given to treatment sections are the integral treatment variable being tested against a business-as-usual control.
    • "On a weekly basis teachers were allocated the same amount of teaching time in treatment and control sections to cover the proscribed curricular content—five 40-min weekly sessions." (p. 60)
    • Relevant Quotes: 1) "On a weekly basis teachers were allocated the same amount of teaching time in treatment and control sections to cover the proscribed curricular content—five 40-min weekly sessions." (p. 60) 2) "In the QuEST 2 treatment sections, teachers used the district textbook, workbook, and district labs, but had access to two additional intervention components: (a) a researcher- designed curriculum that consisted of inquiry- based science lessons, explicitly addressing the academic language needs of ELLs and English proficient students, and (b) professional development to help teachers use the lessons." (p. 60) 3) "There were no other curricular materials provided for use during control sections." (p. 60) 4) "The treatment curriculum was aligned with the Texas and district sixth-grade science standards, covered the same science content as was taught in the district's sixth-grade classrooms, and used the same textbook that was used in the regular classrooms." (p. 60) Detailed Analysis: Applying the criterion B decision procedure: the additional resources given to the treatment group are the researcher-designed supplementary curriculum materials and the associated teacher professional development/mentoring. Critically, no extra student instructional TIME or extra budget was allocated to the treatment condition—both treatment and control sections received "the same amount of teaching time" (five 40-minute weekly sessions) covering the same curricular content and textbook. The additional materials and PD are not a separable, optional add-on incidental to the design; they are explicitly the treatment package being tested (QuEST 2 = supplementary materials + professional development, as stated in the Abstract: "The intervention consisted of supplementary instructional materials and professional development"). The control group received standard "business as usual" district instruction with no special materials, which is the appropriate comparator when the treatment variable itself is the provision of better curriculum materials and teacher training, with time held constant. Criterion B is met because student instructional time and curricular content were held equal across arms, and the additional materials/PD given to the treatment group are integral, explicitly-named components of the treatment package being tested, compared against a business-as-usual control.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study by a different research team was found in the paper or via internet search; all related follow-up work is by overlapping members of the same author team.
    • "The study follows up on a previous study conducted by the authors (August, Branum-Martin, Cardenas-Hagan, & Francis, 2009)..." (p. 55)
    • Relevant Quotes: 1) "The study follows up on a previous study conducted by the authors (August, Branum-Martin, Cardenas-Hagan, & Francis, 2009) but includes more schools and teachers and is focused on different science content, because the district science scope and sequence had changed between the first and second study." (p. 55) 2) "The goal of the second experimental study (August et al., 2009) was to assess the effectiveness of an intervention designed to develop the science knowledge and academic language of middle grades ELLs and their English proficient classmates in five middle schools in south Texas." (p. 56) 3) "In subsequent studies conducted by the authors (in preparation) that were effective in developing ELLs' and English proficient students' academic language and science knowledge, there were 2 days of training prior to the intervention..." (p. 78) Internet Search Findings: A citation search (OpenAlex/Semantic Scholar, 64 citing works checked) found no independent replication of this specific QuEST 2 trial by a different research team. The only closely related follow-up work located was authored by overlapping members of the same author team: August, D., Branum-Martin, L., Duguay, A. L., Powell, J. J., Carlson, C. D., Francis, D. J., & Echevarria, J. (2025). "The impact of an instructional intervention on the science knowledge and academic language of middle-grade English as additional language (EAL) learners and their English proficient classmates." International Journal of Science Education, 48(4), 559-579. This 2025 study is a new school-level RCT/quasi-experimental trial with a new sample of nine schools over two years, not an independent replication of the present QuEST 2 trial's methods and findings. A related 2023 paper by the same team, "A promising intervention designed to improve EAL learners' mathematics skills and associated academic language" (August et al., 2023), is likewise a distinct mathematics intervention with a new sample, not a replication of QuEST 2. Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different research team in a different context, published in a peer-reviewed outlet. The paper itself only references a prior study by the same authors and mentions planned "subsequent studies" also by the same authors. The internet citation search corroborates this: all closely related follow-up work located was authored by overlapping members of the original QuEST research team, not by an independent group, and no publication was found that attempts to reproduce this specific QuEST 2 trial's design and results. Criterion R is not met because no independent replication of this specific study by a different research team was found either in the paper or via internet search.
  • A

    All-subject Exams

    • Only vocabulary/academic language and science were assessed; no other core subjects (e.g. mathematics, social studies) were measured.
    • "Treatment effects were tested separately for academic language and science knowledge." (p. 54)
    • Relevant Quotes: 1) "Treatment effects were tested separately for academic language and science knowledge." (p. 54, Abstract) 2) "In addition, students were assessed on academic language and science knowledge using researcher-developed assessments aligned with the district's sixth-grade science curriculum covered during the 15 weeks the intervention was in place." (p. 63) 3) "All students in the 60 science sections were assessed using the word meaning subtest of the ... GRADE ... prior to and following the intervention." (p. 63) Detailed Analysis: The study's outcome measures cover only two domains: academic-language/vocabulary (via the researcher-developed CBM measure and the GRADE) and science knowledge (via a researcher-developed test). No other core middle-school subjects (e.g., mathematics, social studies, general language arts/writing beyond vocabulary) were assessed. In addition, the science outcome measure itself—central to the "all subjects" requirement since science is the target subject—is a researcher-assembled test rather than a standard, widely recognised standardised exam, so even the assessed subjects are not fully standardised across the board. There is no exception rationale offered (e.g., specialised vocational focus) that would justify limiting assessment to only these two domains for general-education sixth graders. Criterion A is not met because only vocabulary/ academic language and science were assessed, with no coverage of other core subjects such as mathematics or social studies, and the science measure itself was not a standardised exam.
  • G

    Graduation Tracking

    • Criterion Y is not met, and no follow-up publication tracking this study's cohort toward graduation was found via internet search.
    • Relevant Quotes: 1) "The academic vocabulary and science tests were administered prior to and following instruction in each topic area (i.e., Earth and Space Systems, Living Systems and Environment)." (p. 65) 2) "All students in the 60 science sections were assessed using the word meaning subtest of the ... GRADE ... prior to and following the intervention." (p. 63) 3) "In subsequent studies conducted by the authors (in preparation) that were effective in developing ELLs' and English proficient students' academic language and science knowledge, there were 2 days of training prior to the intervention..." (p. 78) Internet Search Findings: A citation search for later publications by the same author team (August, Branum-Martin, Cardenas-Hagan, Francis, Powell) located two subsequent intervention studies: August, D. et al. (2023), "A promising intervention designed to improve EAL learners' mathematics skills and associated academic language," and August, D., Branum-Martin, L., Duguay, A. L., Powell, J. J., Carlson, C. D., Francis, D. J., & Echevarria, J. (2025), "The impact of an instructional intervention on the science knowledge and academic language of middle-grade English as additional language (EAL) learners and their English proficient classmates," International Journal of Science Education, 48(4), 559-579. Both are new intervention trials with new samples of students and schools (the 2025 study involves nine schools over two years), not longitudinal follow-ups tracking the original 1,309 QuEST 2 students. No paper was found that follows the original QuEST 2 cohort beyond the immediate post-test. Detailed Analysis: Per the ERCT standard, if criterion Y (Year Duration) is not met, criterion G cannot be met. Criterion Y is not met here because outcome tracking spanned only about 15 weeks, well short of 75% of an academic year. In addition, neither the original paper nor the located subsequent publications by the same author team describe follow-up of this specific cohort of students toward graduation; the later papers are independent intervention studies with new samples rather than continued tracking of the original participants. Criterion G is not met because criterion Y is not met, and no evidence of graduation tracking of this study's cohort was found in the paper or in subsequent publications by the authors.
  • P

    Pre-Registered

    • No mention of a pre-registered protocol, registry identifier, or registration date is found anywhere in the paper, and internet search found no registry record either.
    • Relevant Quotes: (No quotes found: the paper contains no mention of a study registry, registration identifier, or registration date anywhere in the Methods, Results, or Discussion sections.) Internet Search Findings: A search of publicly available trial/study registries and IES-funded research listings did not locate a pre-registration record for this study (referred to in the literature as "QuEST 2" or "Quality English and Science Teaching 2"). No registration entry, protocol document, or registration date was found. Detailed Analysis: A full-text review of the Methods and surrounding sections (Research Context and Participants, Instructional Intervention, Assessment Instruments, Data Collection, Data Analysis) reveals no reference to pre- registration of the study's hypotheses, design, or analysis plan on any public registry (e.g., ClinicalTrials.gov, ISRCTN, or an educational- research equivalent), nor any registration date that could be checked against the start of data collection. Internet search likewise found no external registry record for this study. Criterion P is not met because no evidence of a pre-registered protocol is present anywhere in the paper or via internet search.