Effects of Academic Vocabulary Instruction for Linguistically Diverse Adolescents: Evidence From a Randomized Field Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the teacher level, with all of each teacher's sections assigned together, exceeding the requirement for class-level randomization.
    • "The cluster-randomized trial meant that each of these 50 teachers was assigned to either the treatment or control condition..." (p. 1166)
    • Relevant Quotes: 1) "This cluster-randomized field trial was conducted in 14 middle schools in a large, urban school district in California that serves an economically and linguistically diverse student population. Fifty teachers and 2,082 students participated. The cluster-randomized trial meant that each of these 50 teachers was assigned to either the treatment or control condition (the method for random assignment is described below in the Study Design section)." (p. 1166) 2) "The cluster-randomized trial required that each of the 50 teachers be assigned to the treatment or control condition. Within each school, teachers were blocked on achievement based on the classroom mean score on the state standards test in ELA administered at the end of the prior academic year, and then randomized within blocks..." (p. 1167) 3) "Twenty-five teachers along with their students in 39 sections were assigned to the control condition and 25 teachers along with their students in 37 sections were assigned to the treatment condition." (p. 1166) Detailed Analysis: The unit of randomization was the teacher, not individual students within a shared classroom. Because every section (class) taught by a given teacher moved together as a block to either treatment or control, there is no within-classroom mixing of conditions, and thus no risk of the contamination this criterion is designed to prevent. This is at least as strong as class-level randomization, satisfying the requirement even though it stops short of the stronger school-level design (each school contained both treatment and control teachers). Criterion C is met because randomization was conducted at the teacher/class-cluster level, with whole teachers (and their sections) assigned to a single condition.
  • E

    Exam-based Assessment

    • The study administered and reported results for several widely used, standardized, norm-referenced tests alongside researcher-created measures.
    • "Gates-MacGinitie Reading Vocabulary Test... a standardized, norm-referenced test, is a widely used assessment of grade-level reading vocabulary knowledge." (p. 1170)
    • Relevant Quotes: 1) "We administered a battery of standardized and researcher-created assessments in the domains of vocabulary, morphological awareness, reading comprehension, and writing to students at pretest (October 2008) and posttest (May 2009). At pretest, the battery included three standardized measures and two researcher-developed measures, and at posttest the battery included two standardized measures and six researcher-developed measures." (p. 1170) 2) "Gates-MacGinitie Reading Vocabulary Test... a standardized, norm-referenced test, is a widely used assessment of grade-level reading vocabulary knowledge. The publisher provides evidence of adequate reliability... and extensive evidence of validity." (p. 1170) 3) "Gates-MacGinitie Reading Test, Fourth Edition: Reading Comprehension... is a standardized, norm-referenced measure... a widely used assessment of students' global reading comprehension..." (p. 1173) 4) "Oral and Written Language Scales: Written Expression... a standardized, norm-referenced measure, is an assessment of students' written language skills..." (p. 1173) 5) "The main effect of treatment was... not significant for the standardized measures of reading comprehension (d = 0.04, p = .4256). The main effect of treatment was also significant for the standardized measure of written expression (d = 0.19, p = .0388)." (p. 1180) Detailed Analysis: Although the study also used several researcher-created instruments (e.g., Academic Word Mastery, Word Association, Morphological Decomposition), it independently administered and reported treatment effects on genuine, publisher- normed standardized tests: the Gates-MacGinitie Reading Vocabulary Test, the Gates-MacGinitie Reading Comprehension test, and the OWLS Written Expression scale. These are widely recognized, validated instruments used outside the context of this specific study, not tests custom-built to match the intervention's content. Criterion E is met because standardized, widely recognized exam-based assessments were used and reported as outcome measures.
  • T

    Term Duration

    • Outcomes were measured from October 2008 to May 2009, roughly seven months after intervention start, well beyond one academic term.
    • "...to students at pretest (October 2008) and posttest (May 2009)."
    • Relevant Quotes: 1) "We administered a battery of standardized and researcher-created assessments... to students at pretest (October 2008) and posttest (May 2009)." (p. 1170) 2) "The 20-week intervention was delivered for 45 minutes a day by randomly assigned ELA teachers over the course of one academic year." (p. 1165) 3) "The program was 20 weeks in length, featuring nine 2-week units... implemented in the context of the participating schools' ELA block, which lasted between 90 and 120 minutes a day." (p. 1168) Detailed Analysis: The interval between the intervention's start (documented via an October 2008 pretest) and the primary outcome measurement (May 2009 posttest) is approximately seven months, considerably longer than the minimum one academic term (roughly 3-4 months) required by this criterion. Criterion T is met because the tracked interval from intervention start to measurement substantially exceeds a full academic term.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline scores, and business-as-usual instruction are described in detail throughout the Method and Results sections.
    • "There were 971 students in the treatment group (72% LM, n = 700) and 1,111 students in the control group (69% LM, n = 768)." (p. 1166)
    • Relevant Quotes: 1) "There were 971 students in the treatment group (72% LM, n = 700) and 1,111 students in the control group (69% LM, n = 768)." (p. 1166) 2) "Since the intervention was designed as an alternative strategy for regular instruction, the amount of time for ELA instruction and the general skills to be taught (i.e., reading, writing, listening, speaking) were comparable across the control and treatment conditions." (p. 1168) 3) "Classroom observations in control classrooms. Control teachers were observed once prior to the start of the intervention period and five times during the intervention period. These observations lasted for 45 minutes and were divided into three 15-minute intervals, with support from an observation protocol featuring 11 categories for curricular content..." (p. 1175) 4) "Analyses conducted on instructional quality prior to estimating treatment impacts showed that the control teachers were rated as moderately higher than the treatment teachers in general instructional quality (d = 0.55)..." (p. 1178) 5) Table 1 reports pretest and posttest means and standard deviations for the control group (overall, and separately for LM and EO subgroups) across all eight outcome measures. (p. 1179) Detailed Analysis: The paper documents the control group's size, language-background composition, baseline (pretest) performance on all outcome measures, and confirms that control teachers continued regular business-as-usual ELA instruction, with their classrooms independently observed and rated for instructional quality across the study period. Criterion D is met because the control group is described in substantial demographic, baseline, and procedural detail.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the teacher level within each school (blocked by school), not at the level of whole schools.
    • "Within each school, teachers were blocked on achievement... and then randomized within blocks..." (p. 1167)
    • Relevant Quotes: 1) "Within each school, teachers were blocked on achievement based on the classroom mean score on the state standards test in ELA administered at the end of the prior academic year, and then randomized within blocks to improve the precision of estimates of treatment effects." (p. 1167) 2) "This cluster-randomized field trial was conducted in 14 middle schools... The cluster- randomized trial meant that each of these 50 teachers was assigned to either the treatment or control condition." (p. 1166) Detailed Analysis: Randomization was explicitly nested within each of the 14 schools, meaning every participating school contained both treatment and control teachers side by side; whole schools were never assigned entirely to one condition. This is a teacher/class-level design, not the stronger school-level design required by this criterion. Criterion S is not met because randomization occurred within schools at the teacher level, not between schools.
  • I

    Independent Conduct

    • The same research team that designed the ALIAS intervention also trained the program specialists, supervised data collection, and conducted the analysis; no independent evaluator is described.
    • "To support the implementation of the intervention, teachers in the treatment group engaged in monthly meetings with one of two program specialists, both former teachers... trained by the research team." (p. 1169)
    • Relevant Quotes: 1) "Academic Language Instruction for All Students (ALIAS) integrates the instructional principles featured in scholarship on vocabulary, reading comprehension, and writing development and instruction. As described, we operationalized these general principles to design a rigorous and engaging approach to academic vocabulary instruction..." (p. 1168) 2) "To support the implementation of the intervention, teachers in the treatment group engaged in monthly meetings with one of two program specialists, both former teachers with extensive experience in the district and trained by the research team." (p. 1169) 3) "Trained research assistants scored student responses using the detailed guide provided in the test manual." (p. 1174) 4) "This research was supported by a grant from the Institute of Education Sciences... and a William T. Grant Foundation Scholars Award, both awarded to Nonie K. Lesaux." (p. 1188) Detailed Analysis: The paper gives no indication that an external, third-party organization independently conducted data collection or analysis. Instead, the same authors who designed ALIAS also trained the program specialists who supported treatment teachers, and the study's own research assistants administered and scored assessments and conducted classroom observations. There is no statement of an independent evaluation agency or blinded external analyst overseeing the trial. Criterion I is not met because the intervention designers and the study's implementers/evaluators are the same research team.
  • Y

    Year Duration

    • The pretest-to-posttest interval (October 2008 to May 2009) covers roughly 75-85% of a typical academic year, and the paper describes the program as delivered over the course of one academic year.
    • "The 20-week intervention was delivered for 45 minutes a day by randomly assigned ELA teachers over the course of one academic year." (p. 1165)
    • Relevant Quotes: 1) "The 20-week intervention was delivered for 45 minutes a day by randomly assigned ELA teachers over the course of one academic year." (p. 1165) 2) "We administered a battery of standardized and researcher-created assessments... to students at pretest (October 2008) and posttest (May 2009)." (p. 1170) Detailed Analysis: The measurement interval runs from an October 2008 pretest to a May 2009 posttest, approximately seven to eight months. A typical U.S. academic year spans roughly September through June (about 9-10 months), so this interval covers on the order of 75-85% of the academic year, at or above the 75% threshold the standard allows for minor shortfalls. The authors additionally frame the intervention as delivered "over the course of one academic year." Criterion Y is met, though the margin above the 75% threshold is modest and worth flagging for careful review.
  • B

    Balanced Control Group

    • Core ELA instructional time was explicitly held comparable across conditions; the additional teacher support the treatment group received is integral to implementing the specific curriculum being tested.
    • "Since the intervention was designed as an alternative strategy for regular instruction, the amount of time for ELA instruction and the general skills to be taught... were comparable across the control and treatment conditions." (p. 1168)
    • Relevant Quotes: 1) "Since the intervention was designed as an alternative strategy for regular instruction, the amount of time for ELA instruction and the general skills to be taught (i.e., reading, writing, listening, speaking) were comparable across the control and treatment conditions." (p. 1168) 2) "These 45-minute lessons were implemented in the context of the participating schools' ELA block, which lasted between 90 and 120 minutes a day." (p. 1168) 3) "To support the implementation of the intervention, teachers in the treatment group engaged in monthly meetings with one of two program specialists... In addition, each teacher in the treatment group had access to a password-protected website, which included electronic versions of the instructional materials and video clips..." (p. 1169) 4) "Analyses conducted on instructional quality prior to estimating treatment impacts showed that the control teachers were rated as moderately higher than the treatment teachers in general instructional quality (d = 0.55), suggesting the evaluation represents a conservative test of the treatment." (p. 1178) Detailed Analysis: Applying the decision procedure: the intervention did add extra teacher-facing resources (monthly coaching meetings with program specialists, a password-protected materials website), which the control-condition teachers did not receive. However, the student-facing instructional time and the general skills targeted (reading, writing, listening, speaking) were explicitly matched across conditions, since ALIAS was designed as a replacement for, not an addition to, regular ELA class time. The extra professional-development support for treatment teachers is not a separable, confounding add-on but an integral component of training teachers to deliver the specific ALIAS curriculum under evaluation, analogous to teacher training resources considered integral in other accepted cases (see ERCT spec examples on teacher professional development as part of the treatment package). There is no evidence that the control group's core educational time or budget was reduced relative to the treatment group; if anything, control teachers were rated somewhat higher on general instructional quality, indicating no systematic disadvantage to the control condition. Criterion B is met because student instructional time was balanced across conditions, and the additional teacher support given to the treatment group is integral to delivering the tested curriculum rather than an unmatched supplementary resource.

• Level 3 Criteria

  • R

    Reproduced

    • The paper contains no reference to an independent replication of this specific trial by another research team, and none could be located in the wider literature.
    • Detailed Analysis: No quotes in the paper reference a separate, independent replication of the ALIAS academic vocabulary intervention or of this specific trial's findings by a different research team. The paper cites related but distinct studies (e.g., Kim et al., 2011; Vaughn et al., 2011) as external evidence for similar design principles in reading and writing interventions, but these are not replications of this study or of ALIAS itself. Internet Search: A search of the wider literature (via OpenAlex and Semantic Scholar citation records for this paper, DOI 10.3102/0002831214532165, which has been cited by roughly 185-200 subsequent works) was conducted to look for an independent replication of ALIAS by a different research team. The citing works located cover related but distinct topics (e.g., morphological awareness interventions, ELL professional development, cross-language morphological analysis), none of which constitutes an independent replication of this specific trial or intervention. No such replication could be identified. Criterion R is not met because no independent replication of this study or of the ALIAS intervention by another research team was found in the paper or in the wider literature.
  • A

    All-subject Exams

    • Outcomes were assessed only in vocabulary, morphology, reading comprehension, and writing (all within English language arts), with no assessment of other core subjects such as mathematics or science.
    • "We administered a battery of standardized and researcher-created assessments in the domains of vocabulary, morphological awareness, reading comprehension, and writing..." (p. 1170)
    • Relevant Quotes: 1) "We administered a battery of standardized and researcher-created assessments in the domains of vocabulary, morphological awareness, reading comprehension, and writing to students at pretest (October 2008) and posttest (May 2009)." (p. 1170) 2) "The 20-week intervention was delivered for 45 minutes a day by randomly assigned ELA teachers... implemented under typical conditions as part of the core English language arts (ELA) instruction..." (p. 1165, p. 1161) Detailed Analysis: All outcome measures used in this study fall within the single subject area of English language arts/literacy (vocabulary, morphology, reading comprehension, writing). No standardized assessments of other core subjects taught in these middle schools, such as mathematics or science, were administered or reported. The paper does not present a rationale tying this to a specialized upper- secondary or vocational exception; sixth-grade ELA instruction is a core, not specialized, subject. Criterion A is not met because the study did not assess performance across all main subjects, only within English language arts.
  • G

    Graduation Tracking

    • The authors explicitly state that outcomes were measured only immediately after the intervention and that longer-term follow-up was not conducted; no later graduation-tracking paper by these authors on this cohort was located.
    • "...we did demonstrate short-term program effects on specific measures of vocabulary and reading comprehension administered immediately after the 20-week intervention, further research is needed to determine whether there are long-term effects of the program." (p. 1188)
    • Relevant Quotes: 1) "In addition, while we did demonstrate short-term program effects on specific measures of vocabulary and reading comprehension administered immediately after the 20-week intervention, further research is needed to determine whether there are long-term effects of the program." (p. 1188) 2) "...to students at pretest (October 2008) and posttest (May 2009)." (p. 1170) Detailed Analysis: The study's only outcome measurement occurred at a posttest immediately following the 20-week intervention (May 2009), with the authors explicitly flagging the absence of any longer-term follow-up as a limitation and direction for future research. There is no indication that students were tracked through to the end of middle school, let alone through graduation from any educational stage. Internet Search: Given criterion Y is met, a search was conducted for any subsequent paper by Lesaux, Kieffer, Kelley, or Harris that follows the same 2008-2009 cohort of sixth graders (N = 2,082, 14 urban California middle schools) into later grades or to graduation. Using OpenAlex and Semantic Scholar citation and author search, no such longitudinal follow-up publication tracking this specific ALIAS cohort could be located. Available later works by these authors (e.g., on morphological awareness and cross-language reading comprehension) involve different populations or analyses of the same posttest data rather than extended tracking of this cohort toward graduation. Criterion G is not met because tracking stopped at the immediate posttest, with no follow-up toward graduation found in this paper or in subsequent publications by the authors.
  • P

    Pre-Registered

    • No statement of pre-registration, a registry platform, or a registration date appears anywhere in the paper, and no matching pre-registration record was located online.
    • Detailed Analysis: A review of the Method, Note, and References sections reveals no mention of a study registry (e.g., a trials registry or an equivalent education research registry), no registration ID, and no statement that hypotheses or analysis plans were published before data collection began (which took place starting October 2008). The Note section credits funding sources (Institute of Education Sciences, William T. Grant Foundation) but does not reference a pre-registered protocol. Internet Search: A search for a matching pre-registration record for this trial (e.g., on registries such as the AEA Social Science Registry or OSF Registries) did not locate any entry for this study or the ALIAS intervention. This is consistent with the study's timeline: data collection began in October 2008, which predates the widespread adoption of pre-registration practices and registries in education research (e.g., the AEA RCT Registry launched in 2013, and OSF Registries became widely used from 2011 onward). Criterion P is not met because no evidence of pre-registration is provided in the paper or found in external registries.