Effectiveness of Supplemental Kindergarten Vocabulary Instruction for English Learners: A Randomized Study of Immediate and Longer-Term Effects of Two Approaches

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the small-group (2-3 student) level within classrooms rather than at the class level, but the intervention is a pull-out, small-group tutoring program delivered outside the regular classroom, which falls under the ERCT exception for personal/tutoring-style teaching.
    • "For both cohorts, students were randomly assigned, within classroom, to small groups of two to three students... Small groups were randomly assigned to conditions within classroom..." (p. 496-497)
    • Relevant Quotes: 1) "A two-cohort cluster-randomized trial was conducted to estimate effects of small-group supplemental vocabulary instruction for at-risk kindergarten English learners (ELs)." (Abstract, p. 490) 2) "For both cohorts, students were randomly assigned, within classroom, to small groups of two to three students, with group sizes based on eligible students within each classroom. Small groups were randomly assigned to conditions within classroom, with the added constraint that classrooms with an uneven number of small groups would have an 'extra' group randomly assigned to one or the other of the conditions." (p. 496-497) 3) "Children in both conditions received small-group instruction outside their classroom for 30 min per day, four days per week, for an average of 20 weeks." (p. 497) 4) "All tutors were recruited from their school communities and hired as district employees paid by the schools with funds provided by the research grant." (p. 499-500) Detailed Analysis: The unit of randomization here is the small group (dyads/triads of 2-3 students) within a classroom, not the class or school as a whole. Under a strict reading of criterion C this would fail, since contamination between conditions could occur among children in the same classroom. However, the ERCT standard includes an explicit exception: if an intervention is designed for personal teaching such as tutoring, student-level (or small-group-level) randomization is acceptable. The intervention studied here is explicitly a pull-out, small-group tutoring program: trained "tutors" (paraeducators) deliver individualized, small-group (2-3 student) lessons outside the regular classroom for 30 minutes per day. This is materially closer to one-to-one/small-group tutoring than to whole-class instruction, and the pull-out delivery outside the classroom substantially reduces opportunity for cross-condition contamination relative to within-classroom instruction. Given this is a personal/tutoring-style intervention, the exception applies. Criterion C is met because, although randomization occurred below the class level, the intervention is a pull-out small-group tutoring program that qualifies for the ERCT tutoring exception.
  • E

    Exam-based Assessment

    • Core outcomes (receptive vocabulary, decoding, spelling) were measured with widely used, standardized norm-referenced tests, satisfying the exam-based assessment requirement even though one supplementary curriculum-based measure was researcher-developed.
    • "Receptive vocabulary was measured using the Peabody Picture Vocabulary Test-IIIA (PPVT-IIIA; Dunn & Dunn, 2006)." (p. 501-502)
    • Relevant Quotes: 1) "Receptive vocabulary was measured using the Peabody Picture Vocabulary Test-IIIA (PPVT-IIIA; Dunn & Dunn, 2006)... Test manual split-half reliability is 0.76 for five-year-olds and .80 for six-year-olds." (p. 501-502) 2) "Decoding was measured using the Woodcock Reading Mastery Test-Revised/Norm Referenced (WRMT-R/NU; Woodcock, 1987/1998) Word Attack subtest... Test manual split-half reliability for kindergartners is 0.94." (p. 502) 3) "Spelling was measured at pretest and posttest using the Wide Range Achievement Test-4 (WRAT-4; Wilkinson & Robertson, 2006) Spelling subtest... Test manual internal consistency reliability coefficient for kindergarteners is .94." (p. 502) 4) "Reading vocabulary was assessed using an experimenter-developed 25-item curriculum-based measure (CBM) of target word reading vocabulary." (p. 502) Detailed Analysis: Three of the four outcome measures used in this study are widely recognized, standardized, norm-referenced instruments with published reliability data: the PPVT-IIIA (receptive vocabulary), the WRMT-R/NU Word Attack subtest (decoding), and the WRAT-4 Spelling subtest (spelling). Only the reading-vocabulary measure (a curriculum-based measure of taught root words) was experimenter-developed and specific to this study's word corpus. Because the study relies substantially on multiple standard, validated psychometric instruments for its principal outcome domains (not solely a custom instrument), the exam-based assessment criterion is satisfied. Criterion E is met because the study used multiple widely recognized standardized tests (PPVT-IIIA, WRMT-R/NU, WRAT-4) alongside one supplementary custom measure.
  • T

    Term Duration

    • Outcomes were measured about seven months after the intervention began (fall pretest to spring posttest), exceeding the minimum one-term requirement, and this is reinforced by the stronger Year Duration criterion also being met.
    • "students were pretested in the fall prior to treatment and posttested in the spring just after treatment, approximately seven months apart, during their kindergarten year" (p. 501)
    • Relevant Quotes: 1) "Children in both conditions received small-group instruction outside their classroom for 30 min per day, four days per week, for an average of 20 weeks." (p. 497) 2) "For each cohort, students were pretested in the fall prior to treatment and posttested in the spring just after treatment, approximately seven months apart, during their kindergarten year, and were followed up approximately seven months after posttest, in the winter of the following year." (p. 501) Detailed Analysis: The immediate outcome measurement (posttest) occurred approximately seven months after the pretest/start of the intervention period, which exceeds the minimum one-academic-term (3-4 month) requirement. In addition, the study also includes a longer-term follow-up roughly 14 months after the intervention began, which independently satisfies the stronger Year Duration criterion (see criterion Y), and per the ERCT standard, meeting Y automatically satisfies T. Criterion T is met because the measurement interval from intervention start to posttest (about seven months) exceeds one academic term, and is further reinforced by the year-long follow-up tracking.
  • D

    Documented Control Group

    • Although this is a two-active-treatment comparison rather than a treatment-vs-no-treatment design, the comparison (IBR) condition is extensively documented, including demographics, sample sizes, and detailed procedures.
    • "Table 1. Disaggregated student demographic characteristics" showing Connections and IBR groups separately by gender, SPED/ELL services, language proficiency level, and home language. (p. 498)
    • Relevant Quotes: 1) "Table 1. Disaggregated student demographic characteristics" comparing Connections (n = 163) and IBR (n = 161) groups on gender, SPED services, ELL services, state language test level, and home language. (p. 498) 2) "Students assigned to the IBR condition received instruction in the same target vocabulary provided in the Connections condition. Instruction was provided in the context of reading aloud a storybook..." (p. 500) 3) "...163 students in 75 Connections small groups and 161 students in 72 IBR small groups." (p. 497), with IBR procedures described in full in Table A4, Appendix A (p. 526) Detailed Analysis: This study does not include a business-as-usual, no-treatment control group; instead it compares two active supplemental treatments (Connections vs. IBR), a limitation the authors themselves acknowledge ("difficult to interpret these gains without a no-treatment control comparison"). Treating IBR as the comparison ("control") arm relative to Connections, the paper provides thorough documentation: sample sizes, demographic breakdowns by condition (Table 1), baseline pretest scores by condition (Table 2), and a full description of the IBR procedures (Table A4). This level of detail allows readers to verify comparability of the comparison group at baseline and to understand exactly what instruction it received. Criterion D is met because the comparison group's characteristics, size, and procedures are documented in detail, even though the design lacks a no-treatment control.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was conducted at the small-group level within classrooms across 13-20 schools, not at the school level, so the stronger school-level RCT criterion is not satisfied.
    • "For both cohorts, students were randomly assigned, within classroom, to small groups of two to three students..." (p. 496)
    • Relevant Quotes: 1) "Sites were public elementary schools located in an urban area of the U.S. Pacific Northwest with large enrollments of English learner (EL) students." (p. 495) 2) "For both cohorts, students were randomly assigned, within classroom, to small groups of two to three students, with group sizes based on eligible students within each classroom. Small groups were randomly assigned to conditions within classroom..." (p. 496-497) 3) "The final combined kindergarten sample included 324 EL students in 127 small groups... Small groups were nested within 27 tutors/tutor teams across 40 classrooms and 13 schools..." (p. 497) Detailed Analysis: Randomization occurred at the small-group level within individual classrooms, not at the level of whole schools. Multiple small groups from the same classroom, and multiple classrooms from the same school, could be assigned to different conditions. This is a substantially weaker unit of randomization than school-level assignment, and does not meet the stronger S criterion, which requires that entire schools (or comparable implementation units) be randomized as a whole. Criterion S is not met because randomization occurred at the small-group/within-classroom level, not at the school level.
  • I

    Independent Conduct

    • The Connections intervention materials were designed by two of this paper's own authors (Nelson and Vadasy), who also led this evaluation, so the study was not conducted independently of the intervention designers.
    • "Connections materials included a 12- by 17-inch manual used to present lessons to small groups (Nelson & Vadasy, 2007)." (p. 499)
    • Relevant Quotes: 1) "Connections materials included a 12- by 17-inch manual used to present lessons to small groups (Nelson & Vadasy, 2007)." (p. 499) 2) "Nelson, J. R., & Vadasy, P. (2007). Early vocabulary connections: First words to know and decode. Longmont, CO: Sopris West." (References, p. 521) 3) "Patricia F. Vadasy, Oregon Research Institute, Seattle, Washington, USA... Elizabeth A. Sanders, University of Washington... J. Ron Nelson, University of Nebraska-Lincoln" (author affiliations, p. 490) 4) "In an earlier study Connections was compared to a similar IBR storybook-reading approach (Nelson, Vadasy, & Sanders, 2011)." (p. 495) Detailed Analysis: The Connections curriculum evaluated in this study was itself authored by two of the present paper's authors, J. Ron Nelson and Patricia F. Vadasy (Nelson & Vadasy, 2007). The same author team also designed and reported the earlier pilot efficacy study of Connections (Nelson, Vadasy, & Sanders, 2011) and now conducts this larger follow-up trial. There is no mention anywhere in the paper of an independent, third-party organization conducting data collection, analysis, or interpretation separate from the intervention's developers. This is the paradigm case the I criterion is designed to catch: the researchers who created the tested materials are also the ones designing, running, and analyzing the evaluation of those same materials. Criterion I is not met because the intervention developers (Nelson and Vadasy) are also the study's lead investigators, with no independent evaluation team involved.
  • Y

    Year Duration

    • Students were tracked to a first-grade follow-up roughly 14 months after the intervention began (kindergarten fall start to first-grade winter follow-up), exceeding the one-academic-year requirement.
    • "were followed up approximately seven months after posttest, in the winter of the following year" (p. 501)
    • Relevant Quotes: 1) "students were pretested in the fall prior to treatment and posttested in the spring just after treatment, approximately seven months apart, during their kindergarten year, and were followed up approximately seven months after posttest, in the winter of the following year." (p. 501) 2) "For each cohort, seven months after kindergarten posttest we follow-up tested all students who had participated in the study and were still enrolled in the school district." (p. 497) 3) "At first-grade follow-up, longer-term gains were again greater for Connections students, but with smaller effect sizes (d = .29 and .27, respectively)." (Abstract, p. 490) Detailed Analysis: The intervention began in the fall of the kindergarten year, with immediate outcomes assessed about seven months later (spring posttest), and a longer-term follow-up assessment conducted a further seven months after that, in the winter of first grade. Summing these intervals, the total tracking period from intervention start to the longer-term follow-up measurement is approximately 14 months, which comfortably exceeds a full academic year (~9-10 months) and thus clears the 75% threshold specified by the ERCT standard. The study's explicit framing around "Immediate and Longer-Term Effects" reflects this extended tracking window. Criterion Y is met because outcomes were tracked to a first-grade follow-up approximately 14 months after the intervention began, exceeding the one-year requirement.
  • B

    Balanced Control Group

    • Both experimental conditions (Connections and IBR) received identical amounts of supplemental instructional time and tutor attention, so resources were balanced between the two arms compared.
    • "Children in both conditions received small-group instruction outside their classroom for 30 min per day, four days per week, for an average of 20 weeks." (p. 497)
    • Relevant Quotes: 1) "Children in both conditions received small-group instruction outside their classroom for 30 min per day, four days per week, for an average of 20 weeks." (p. 497) 2) "Each tutor was trained in both treatments and had at least two small groups each." (p. 500) 3) "The observed student mean was 73.10 days of tutoring out of 80 possible (SD = 7.28, range = 40-80 days; Connections M = 72.82, SD = 7.63; IBR M = 73.39, SD = 6.92)." (p. 501) 4) Table A2 attribute comparison shows both conditions share the same target words, group sizes, tutor training, and overall lesson structure, differing only in specific instructional techniques (decoding/spelling practice vs. storybook reading). (Appendix, p. 524) Detailed Analysis: Applying the ERCT decision process for criterion B (per the updated specification and decision tree): both conditions here receive extra resources relative to ordinary classroom instruction (pull-out tutoring time), so the "no extra resources" branch does not directly apply. This study compares two active treatments rather than a treatment against a business-as-usual control, so the relevant question is whether the two arms received comparable time and resources relative to each other (i.e., whether CONTROL_MATCHES_RESOURCES is satisfied, treating either arm as the comparison condition for this purpose). The evidence shows they did: both conditions received the same dosage (30 minutes/day, four days/week, ~20 weeks), delivered by the same pool of trained tutors, with statistically similar attendance (72.82 vs. 73.39 days out of 80 possible). The only difference between conditions is the specific instructional technique (explicit decoding/spelling instruction vs. storybook reading), not the quantity of time, tutor attention, or session structure, and this instructional-technique contrast is precisely the comparison the study is designed to test. This matches the balanced-active control pattern described in the standard's own worked examples (e.g., comparable dosage/structure between an inference intervention and an active spatial-training control). Criterion B is met because both compared conditions received equal instructional time, tutor attention, and session structure, differing only in instructional technique.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study or the Connections intervention by a different research team was found; the only prior and follow-on studies were conducted by the same overlapping author team, including the "replication" funding itself.
    • "In an earlier study Connections was compared to a similar IBR storybook-reading approach (Nelson, Vadasy, & Sanders, 2011)." (p. 495)
    • Relevant Quotes: 1) "In an earlier study Connections was compared to a similar IBR storybook-reading approach (Nelson, Vadasy, & Sanders, 2011). Kindergarten EL students (from Spanish-speaking homes) across six public schools in the Midwest were randomly assigned..." (p. 495) 2) "In a follow-up study six months after intervention ended, results showed that Connections students were still significantly higher than IBR peers on reading vocabulary and decoding (Vadasy, Nelson, & Sanders, 2013)." (p. 495) 3) "An initial efficacy study of this intervention found positive effects on both word reading and vocabulary. This study is designed to replicate these findings." (IES grant award abstract, R305A130700, "Efficacy of Supplemental Early Vocabulary Connections Instruction for English Language Learners," PI Patricia Vadasy, co-PI J. Ron Nelson, 2011-2013) Detailed Analysis: The only prior and follow-up studies of the Connections intervention cited in the paper (Nelson, Vadasy, & Sanders, 2011; Vadasy, Nelson, & Sanders, 2013) were conducted by the same overlapping author team as the present study (Vadasy, Sanders, and/or Nelson appear on all three). Internet research into the IES funding record for this study (award R305A130700) confirms that this 2015 paper is itself the funded "replication" of the 2011 pilot study, explicitly framed in the grant abstract as designed by the same PI/co-PI team to replicate their own earlier findings, not as an independent verification. Web searches for citing literature and independent trials of the Connections vs. IBR comparison located no independent, peer-reviewed replication of this specific comparison by a different research team. A superficially related paper was found (Coyne, McCoach, Loftus-Rattan, Baker, & Ware, 2022, "Long-Term Effects of Kindergarten Vocabulary Instruction and Intervention on Target Vocabulary Knowledge Through Second Grade," Topics in Language Disorders, 42(4), E9-E28), but it evaluates a different vocabulary intervention design by an unrelated author team and is not a replication of Connections vs. IBR. Because criterion R requires replication by a different, independent research team, self-replication by the original authors does not satisfy it. Criterion R is not met because all related, follow-up, and even the explicitly funded "replication" study of this intervention were conducted by the same author team, and no independent replication by different researchers was found via internet search.
  • A

    All-subject Exams

    • The study measured only vocabulary, decoding, and spelling outcomes (all within the language arts/literacy domain); no other core subjects such as mathematics were assessed.
    • "Receptive vocabulary was measured using the Peabody Picture Vocabulary Test-IIIA... Reading vocabulary was assessed using an experimenter-developed... measure... Decoding was measured using the Woodcock Reading Mastery Test-Revised... Spelling was measured... using the Wide Range Achievement Test-4..." (p. 501-502)
    • Relevant Quotes: 1) "Descriptions of assessments, including sample reliabilities (Cronbach's alpha), are as follows: 1. Receptive vocabulary... 2. Reading vocabulary... 3. Decoding... 4. Spelling..." (p. 501-502) 2) "With four outcomes analyzed (receptive language, reading vocabulary, decoding, and spelling) for each of the two sets of gains, we run the risk of inflating Type I error..." (p. 504) Detailed Analysis: All four outcome measures used in this study (receptive vocabulary, reading vocabulary, decoding, and spelling) fall within a single academic domain: early literacy/language arts. No mathematics, science, or other core-subject outcomes were assessed. The study does not offer any rationale for restricting assessment to literacy-related outcomes as a deliberate specialization exception (e.g., a vocational/highly specialized program); rather, this simply reflects the intervention's exclusive focus on vocabulary and early literacy skills. Since criterion A requires assessment across all main subjects (absent a justified specialization exception), and only literacy-related outcomes were measured, this criterion is not satisfied. Criterion A is not met because only literacy-related outcomes (vocabulary, decoding, spelling) were assessed, with no coverage of other core subjects such as mathematics.
  • G

    Graduation Tracking

    • Tracking ended at a first-grade follow-up, well short of graduation, with no mention of continued tracking or planned follow-up studies beyond this point, and no such follow-up was located via internet search.
    • "we follow-up tested all students who had participated in the study and were still enrolled in the school district" (p. 497)
    • Relevant Quotes: 1) "For each cohort, seven months after kindergarten posttest we follow-up tested all students who had participated in the study and were still enrolled in the school district." (p. 497) 2) "At first-grade follow-up, longer-term gains were again greater for Connections students, but with smaller effect sizes (d = .29 and .27, respectively)." (Abstract, p. 490) 3) "Findings suggest future study of these instructional features for teaching a broader, less-constrained word corpus, and for teaching vocabulary to older students with established word-reading skills." (p. 516, Conclusion) Detailed Analysis: The study's longest follow-up assessment occurs in the winter of first grade, only about 14 months after the kindergarten intervention began. There is no description of tracking students through to graduation from primary school, secondary school, or any other terminal educational milestone, nor is there mention of a planned or published follow-up study extending tracking further. The Conclusion instead points to general future-research directions rather than describing any ongoing or planned graduation-tracking effort for this specific cohort. Internet searches for subsequent publications by Vadasy, Sanders, and/or Nelson that might track this same two-cohort kindergarten sample beyond first grade found no such paper; the associated IES grant record (R305A130700) covers only this study's 2011-2013 funding period and does not describe further follow-up phases. A superficially related paper was located (Coyne, McCoach, Loftus-Rattan, Baker, & Ware, 2022, "Long-Term Effects of Kindergarten Vocabulary Instruction and Intervention on Target Vocabulary Knowledge Through Second Grade," Topics in Language Disorders, 42(4), E9-E28), but this is an independent study by a different research team evaluating a different vocabulary intervention design, not a continuation of the Connections/IBR cohort reported here, so it does not satisfy graduation tracking for this study. Criterion G is not met because student tracking ended at a first-grade follow-up, far short of graduation, with no evidence from the paper or from internet search of further planned or completed long-term tracking of this cohort.
  • P

    Pre-Registered

    • The paper contains no statement or reference indicating that the study protocol was pre-registered before data collection began, and no registry record was located via internet search.
    • Relevant Quotes: No quotes were found anywhere in the paper (methods, procedures, acknowledgments, or references) referencing a trial registry, registration number, or a pre-registered protocol/analysis plan published before data collection began. Detailed Analysis: The paper describes its methodology, recruitment, eligibility, randomization, and analysis plan in detail, but at no point mentions registration with a clinical or educational trial registry (e.g., ClinicalTrials.gov, ISRCTN, AEA RCT Registry, or similar), nor gives any date of pre-registration. Internet searches of the associated IES funding record (R305A130700, "Efficacy of Supplemental Early Vocabulary Connections Instruction for English Language Learners," Patricia Vadasy PI, 2011-2013) found no mention of protocol pre-registration on a trial registry. Given the 2015 publication date and the study design (an education efficacy trial with data collection beginning in 2011), pre-registration was not a described or located part of the study's conduct. Criterion P is not met because there is no evidence in the paper, or from internet search of related funding and registry records, of a pre-registered protocol prior to data collection.