Improving Young English Learners' Language and Literacy Skills Through Teacher Professional Development: A Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation occurred at the school level, which is stronger than and therefore satisfies the class-level RCT requirement.
    • "We matched schools on the percentage of students eligible for free or reduced-price lunch and then randomly assigned each school within a pair to either the intervention group or a waitlist/control condition." (pp. 126-127)
    • Relevant Quotes: 1) "We matched schools on the percentage of students eligible for free or reduced-price lunch and then randomly assigned each school within a pair to either the intervention group or a waitlist/control condition." (pp. 126-127) 2) "Each team of three submitted an application for the study, which included the principal's signature indicating that they understood that teachers in schools randomly assigned to the intervention group would need to collaborate for 30 minutes each week." (p. 126) Detailed Analysis: The paper explicitly states that randomisation occurred at the school level, not at the level of individual students or single classrooms within a school. Under the ERCT Standard, a stronger school-level RCT automatically satisfies the weaker class-level RCT requirement, since randomising whole schools prevents any contamination between treatment and control conditions at the class level. Criterion C is met because the study used school-level randomisation, which is stronger than and therefore satisfies the class-level requirement.
  • E

    Exam-based Assessment

    • The study used the WMLS-R, a widely recognised standardised language and literacy assessment.
    • "Students were assessed using the Woodcock Muñoz Language Survey-Revised Normative Update, (WMLS-R; Schrank, McGrew, & Dailey, 2010), which measures students' language and literacy skills in both English and Spanish." (p. 129)
    • Relevant Quotes: 1) "Students were assessed using the Woodcock Muñoz Language Survey-Revised Normative Update, (WMLS-R; Schrank, McGrew, & Dailey, 2010), which measures students' language and literacy skills in both English and Spanish." (p. 129) 2) "The seven subscales include Picture Vocabulary (median reliability .91), Verbal Analogies (median reliability .90), Letter-Word Identification (median reliability .97), Dictation (median reliability .94), Understanding Directions (median reliability .82), Story Recall (median reliability .76), and Passage Comprehension (median reliability .82)." (p. 129) Detailed Analysis: The WMLS-R is a commercially published, nationally normed, widely used standardized language assessment with documented psychometric properties (median subtest reliabilities reported), not a custom instrument designed solely for this study. It was administered in a standard published form (Form A/Form B), further supporting its status as a recognised standardised exam. Criterion E is met because the study used the WMLS-R, a widely recognised, standardised language and literacy assessment.
  • T

    Term Duration

    • Outcomes were measured at the end of a full school year, far exceeding the one-term minimum.
    • "We observed each teacher three times during the school year and assessed students individually at the beginning and end of the school year using the Woodcock Muñoz Language Survey (WMLS)." (p. 118)
    • Relevant Quotes: 1) "During the 2014–2015 school year, we conducted a pilot study of the DCCS Professional Development Program using a randomized controlled trial." (p. 126) 2) "We observed each teacher three times during the school year and assessed students individually at the beginning and end of the school year using the Woodcock Muñoz Language Survey (WMLS)." (p. 118) Detailed Analysis: Outcomes were measured at the end of the full 2014-2015 school year, having begun at the start of that same school year. This interval far exceeds the minimum of one academic term required by this criterion. Since the stronger Year Duration criterion (Y) is also met, this weaker Term Duration criterion is automatically satisfied as well. Criterion T is met because outcomes were measured a full school year after the intervention began, well beyond one term.
  • D

    Documented Control Group

    • The control group's demographics and baseline characteristics are extensively documented in Tables 2 and 3.
    • "No differences were found between the intervention and the control groups for any of these teacher demographic variables (see Table 2)." (p. 130)
    • Relevant Quotes: 1) "Table 2 presents Classroom and English as a Second Language (ESL) Teacher Demographics ... Control n = 16 ... Total years of teaching M (SD) 10.23 (7.79) ... Highest degree completed ... Licensure ... Language proficiency" (Table 2, p. 128) 2) "Table 3 presents Student Characteristics ... Control n = 41 ... Gender ... Grade ... Age in years M (SD) 6.66 (.87) ... Pretest Broad English Ability Total W score M (SD) 438.68 (30.90) ... Free or reduced lunch" (Table 3, p. 130) 3) "No differences were found between the intervention and the control groups for any of these teacher demographic variables (see Table 2)." (p. 130) 4) "The students in the two groups did not differ on any of the other demographic variables (see Table 3)." (p. 130) Detailed Analysis: The paper provides detailed demographic and baseline documentation of the control group at both the teacher level (Table 2: gender, years of teaching, degree, licensure, ESL certification, SIOP training, language proficiency) and the student level (Table 3: gender, grade, age, pretest English and Spanish ability, attendance, free/reduced lunch status). The authors also explicitly compare control and intervention groups on these variables and report where they are balanced or differ (e.g., significantly more males in the control group). Criterion D is met because the control group's composition, size, and baseline characteristics are extensively documented in Tables 2 and 3.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was conducted at the school level, satisfying the school-level RCT requirement.
    • "We matched schools on the percentage of students eligible for free or reduced-price lunch and then randomly assigned each school within a pair to either the intervention group or a waitlist/control condition." (pp. 126-127)
    • Relevant Quotes: 1) "We matched schools on the percentage of students eligible for free or reduced-price lunch and then randomly assigned each school within a pair to either the intervention group or a waitlist/control condition." (pp. 126-127) 2) "Forty-five teachers from 12 elementary schools within three school districts in the Southeast participated in the study." (p. 127) Detailed Analysis: The unit of randomisation was the entire school: schools were paired based on socioeconomic composition and then one school in each pair was randomly assigned to the intervention or the waitlist/control condition. All ESL and classroom teacher teams within an assigned school shared that school's condition. This matches the ERCT definition of a school-level RCT. Criterion S is met because randomisation was conducted at the level of whole schools, not classes or students within a school.
  • I

    Independent Conduct

    • The same research team that designed the DCCS program also delivered coaching, conducted observations, and analysed the data, with no independent third-party evaluator involved.
    • "A member of the research team who is a former ESL teacher and elementary school principal provided instructional coaching during one of the structured collaboration meetings between the ESL teacher and the classroom teachers at each school." (p. 125)
    • Relevant Quotes: 1) "A member of the research team who is a former ESL teacher and elementary school principal provided instructional coaching during one of the structured collaboration meetings between the ESL teacher and the classroom teachers at each school." (p. 125) 2) "Members of the research team conducted all classroom observations." (p. 129) 3) "Babinski, L. M., Sánchez, M., Knotek, S. E., Amendum, S., & Corra, A. (2013, November). ESL and classroom teachers' views on collaboration." and "Babinski, L. M., Sánchez, M., Amendum, S., & Knotek, S. (2016, May). DCCS: ESL and classroom teachers working together with students and families." (p. 140, References) Detailed Analysis: The same research team (Babinski, Amendum, Knotek, Sánchez) that designed the DCCS professional development program, as evidenced by their prior publications describing its collaboration model and cultural wealth framework, also delivered the ongoing instructional coaching to intervention teachers, conducted all classroom observations used to measure fidelity and outcomes, and analysed the resulting data (with Patrick Malone as the quantitative analyst on the same author team). There is no statement anywhere in the paper indicating that data collection, coaching, or analysis were carried out by an independent, third-party organisation unaffiliated with the program's designers. Criterion I is not met because the developers of the intervention also delivered the coaching, conducted the observations, and analysed the data, with no independent third-party oversight described.
  • Y

    Year Duration

    • The intervention and outcome measurement spanned a complete academic school year.
    • "During the 2014–2015 school year, we conducted a pilot study of the DCCS Professional Development Program using a randomized controlled trial." (p. 126)
    • Relevant Quotes: 1) "During the 2014–2015 school year, we conducted a pilot study of the DCCS Professional Development Program using a randomized controlled trial." (p. 126) 2) "We observed each teacher three times during the school year and assessed students individually at the beginning and end of the school year using the Woodcock Muñoz Language Survey (WMLS)." (p. 118) Detailed Analysis: The intervention and its measurement spanned the entirety of the 2014-2015 academic school year, with pretest assessment at the start of the year and posttest assessment at the end of the same school year. This fully satisfies the requirement of tracking outcomes for at least 75% of one academic year. Criterion Y is met because outcomes were tracked across a complete academic school year.
  • B

    Balanced Control Group

    • The additional professional development time is the explicit treatment variable under study, so the business-as-usual control group does not violate this criterion.
    • "Research Question 1: Does participation in the professional development program increase ESL and classroom teachers' use of high-impact instructional strategies as compared to teachers in the control group?" (p. 126)
    • Relevant Quotes: 1) "Using a randomized controlled trial, we tested a new teacher professional development program for increasing the language and literacy skills of young Latino English learners..." (pp. 117-118) 2) "Teachers attended 35 hours of professional development during the Summer Institute and an additional 6 hours during the modules." (p. 126) 3) "Research Question 1: Does participation in the professional development program increase ESL and classroom teachers' use of high-impact instructional strategies as compared to teachers in the control group?" (p. 126) 4) "...randomly assigned each school within a pair to either the intervention group or a waitlist/control condition." (pp. 126-127) Detailed Analysis: Applying the updated Criterion B decision procedure: extra time/budget is present (a five-day, 35-hour Summer Institute, four additional 1.5-hour modules totaling 6 hours, coaching roughly every six weeks, and a structured 30-minutes-per-week collaboration requirement for intervention teachers). These resources are not a negligible difference, and the control group did not receive matched hours. However, the extra teacher time and training is not a supplementary add-on to some other core intervention; it is itself the explicit treatment variable being tested. Research Question 1 directly frames the professional development program's dosage as the causal factor under study ("Does participation in the professional development program increase ... teachers' use of high-impact instructional strategies"). Per the decision tree, when the extra resources are the treatment variable being tested (RESOURCES_ARE_TREATMENT = true), the criterion is met regardless of whether the control group was given matched hours, since the business-as-usual "waitlist/control" condition is the correct comparator by design. This is not a within-subjects design, so the within-subjects branch does not apply. Criterion B is met because the additional teacher time and resources are the explicit treatment variable being tested, with the control group representing an appropriate business-as-usual baseline.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of the DCCS professional development program by a different research team was found; a 2025 follow-up trial exists but was conducted by the same author team.
    • Relevant Quotes: 1) No quotes in the paper mention an independent replication of this specific study or the DCCS program by another research team. 2) Related presentations and reports cited in the references (e.g., "Babinski, L. M., Amendum, S., Knotek, S., & Sánchez, M. (2017, April). Improving young English learners' language and literacy skills through teacher professional development." and "Babinski, L. M., Sánchez, M., Amendum, S., & Knotek, S. (2016, May). DCCS: ESL and classroom teachers working together with students and families.") are all authored by the same original research team, not independent replicators. 3) Internet search identified a subsequent article by the same author team: Babinski, L. M., Amendum, S. J., Carrig, M., Knotek, S. E., & Sánchez, M. (2025). "Teaching young multilingual learners: impacts of a professional learning programme on teachers' practices and students' language and literacy skills." Journal of Multilingual and Multicultural Development. doi:10.1080/ 01434632.2025.2472880. This is a larger RCT of the renamed BELLA (Bridging English Language Learning and Academics) program, previously called DCCS, conducted by the same core research team (Babinski, Amendum, Knotek, Sánchez), not an independent evaluator. 4) Institute of Education Sciences award records show an "Efficacy of the DCCS Program: ESL and Classroom Teachers Working Together with Students and Families" grant (2017-2022) led by the same investigators, again indicating continued work by the original developers rather than independent replication. Detailed Analysis: Criterion R requires that the specific study (or its central intervention and design) be independently replicated by a different research team in a different context and published in a peer-reviewed outlet. A targeted internet search for independent replications of the DCCS/BELLA professional development program found only project pages, grant records, conference presentations, and a 2025 follow-up efficacy trial, all authored or led by the same original DCCS research team (Babinski, Amendum, Knotek, Sánchez), with no evidence of a subsequent peer-reviewed replication conducted by an unaffiliated research team. Criterion R is not met because no independent replication of the DCCS/BELLA program or this study's findings by a different research team was found.
  • A

    All-subject Exams

    • Only language and literacy outcomes were measured; no other core subjects were assessed, and no exception applies.
    • "The seven subscales include Picture Vocabulary ..., Verbal Analogies ..., Letter-Word Identification ..., Dictation ..., Understanding Directions ..., Story Recall ..., and Passage Comprehension ..." (p. 129)
    • Relevant Quotes: 1) "Students were assessed using the Woodcock Muñoz Language Survey-Revised Normative Update, (WMLS-R; Schrank, McGrew, & Dailey, 2010), which measures students' language and literacy skills in both English and Spanish." (p. 129) 2) "The seven subscales include Picture Vocabulary ..., Verbal Analogies ..., Letter-Word Identification ..., Dictation ..., Understanding Directions ..., Story Recall ..., and Passage Comprehension ..." (p. 129) Detailed Analysis: All seven WMLS-R subtests used as outcomes measure aspects of English (and Spanish) language and literacy only. No mathematics, science, or other core subject assessments are reported anywhere in the study. The exception for specialised interventions applies only to highly specialised upper-secondary or vocational programs with a clearly explained rationale; this study involves general K-2 elementary students, not a specialised upper-secondary/vocational context, so the exception does not apply. Criterion A is not met because only language and literacy outcomes were assessed, with no measurement of other core subjects and no applicable exception.
  • G

    Graduation Tracking

    • Tracking ended after one school year, with no follow-up of this cohort through graduation reported or found.
    • "We observed each teacher three times during the school year and assessed students individually at the beginning and end of the school year using the Woodcock Muñoz Language Survey (WMLS)." (p. 118)
    • Relevant Quotes: 1) "We observed each teacher three times during the school year and assessed students individually at the beginning and end of the school year using the Woodcock Muñoz Language Survey (WMLS)." (p. 118) 2) "The next step for the teacher professional development program is a larger randomized controlled efficacy trial with a sufficient number of schools to analyze the data using a nested model to provide evidence of its effectiveness across many different types of school contexts." (p. 139) 3) Internet search located a 2025 follow-up article by the same authors, Babinski, Amendum, Carrig, Knotek, & Sánchez (2025, Journal of Multilingual and Multicultural Development), describing a subsequent, larger RCT of the renamed BELLA program. This follow-up study evaluates a new, larger cohort of teachers and students rather than tracking the original 2014-2015 pilot cohort of 105 students through to graduation. Detailed Analysis: Outcome data collection ended at the close of the single 2014-2015 pilot school year, with no indication anywhere in the paper of continued tracking of the kindergarten, first-, or second-grade students beyond that year. The Discussion and Implications sections describe only a future, larger efficacy trial as next steps, not a follow-up of this cohort through to graduation. A subsequent 2025 publication by the same authors exists, but it reports on a new, larger efficacy trial of the renamed BELLA program rather than continued tracking of the original pilot study's 105 students to graduation. No subsequent published follow-up study tracking this same cohort to graduation was identified. Criterion G is not met because tracking ended after one school year, with no evidence of follow-up of this cohort through graduation.
  • P

    Pre-Registered

    • No pre-registration of the study protocol is referenced in the paper or found elsewhere.
    • Relevant Quotes: 1) No quote anywhere in the paper references a trial registry, registration ID, or a pre-registration date. 2) "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A120290 to Duke University." (p. 140, Notes) Detailed Analysis: The paper mentions its IES grant funding number but does not cite a study pre-registration (such as an OSF, ClinicalTrials.gov, or AEA RCT registry entry) with hypotheses, methods, and planned analyses published before data collection began. A targeted internet search, including a review of the What Works Clearinghouse study summary page for this trial and general web searches for a registry entry linked to this DOI, did not locate any pre-registration record for this study. Criterion P is not met because no evidence of a pre-registered protocol was found in the paper or through further searching.