The Pathway to Academic Success: Scaling Up a Text-Based Analytical Writing Intervention for Latinos and English Learners in Secondary School

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was conducted at the teacher/classroom level (whole focal classes assigned to treatment or control), which satisfies the class-level RCT requirement.
    • "Teachers were randomized together with the students in a single focal class, within school-by-grade blocks to provide better equivalence across baseline indicators of key outcomes and of contextual factors." (p. 21)
    • Relevant Quotes: 1) "Two hundred and thirty teachers from partner districts affiliated with the NWP sites were stratified by school and grade and then randomly assigned to the treatment or control group." (Abstract, p. 1) 2) "Teachers were randomized together with the students in a single focal class, within school-by-grade blocks to provide better equivalence across baseline indicators of key outcomes and of contextual factors." (p. 21) 3) "SRI then randomized approximately half of the recruited teachers (and their classrooms) into treatment and half into control (delayed treatment)." (p. 21) 4) "Randomization occurred within grade-level blocks consisting of school-by-grade groupings, although not all schools had multiple study teachers in the same grade." (pp. 21-22) Detailed Analysis: The unit of randomization was the teacher together with an entire intact focal class of students, not individual students within a classroom. Each teacher was randomized with his or her whole focal class into either the treatment or the control (delayed treatment) condition within school-by-grade blocks. This is precisely the class-level cluster randomization that criterion C requires: whole classes were allocated to conditions, which prevents within-class contamination between treatment and control students. The randomization process is clearly described (performed by the independent evaluator SRI, with blocking by school and grade, 113 treatment and 117 control teachers), so the process is transparent and properly implemented. All quotes were checked against the source PDF and are verbatim. Criterion C is met because entire teachers/classes, not individual students within a class, were randomly assigned to conditions.
  • E

    Exam-based Assessment

    • The study used a recognized state standardized exam (SBAC ELA) as an outcome measure alongside the study-created writing assessment, satisfying the exam-based assessment requirement.
    • "Finally, SRI ran impacts on California's standardized ELA assessment, SBAC." (p. 42)
    • Relevant Quotes: 1) "SRI examined two types of student learning outcomes: one was writing data from study created prompts and the other were standardized assessments." (p. 38) 2) "The student writing was scored with the Analytic Writing Continuum for Literary Analysis (AWC-LA). Over a decade, the NWP developed the Analytic Writing Continuum (AWC), which has been shown to be a valid and reliable measure of student writing (Bang, 2013)." (p. 39) 3) "As an additional measure of student achievement, SRI collected state ELA assessment data as available. California began administering the Smarter Balanced Assessment Consortium (SBAC) assessment in spring 2014 to students in grades 3-8 and 11. These data served as an additional outcome measure for students." (p. 41) 4) "Finally, SRI ran impacts on California's standardized ELA assessment, SBAC." (p. 42) 5) "SRI found no evidence of a transfer of the intervention's impact on writing to a broader measure of student ELA achievement (d=.06, s.e.=.05, p>.10...)." (p. 46) Detailed Analysis: The primary outcome was an on-demand writing assessment built from study-created prompts, scored on the NWP AWC-LA rubric; this instrument, although scored with a validated rubric by blinded external scorers, was designed for the study and is not by itself a widely recognized standardized exam. However, the study also formally measured and reported outcomes on the Smarter Balanced Assessment Consortium (SBAC) test, California's official state standardized ELA assessment, using the prior-year California Standards Test as the baseline. The ERCT standard explicitly states that what matters is that a standard exam-based assessment is used, not whether a positive effect is found on it; the null SBAC result (d=.06, n.s.) therefore does not affect this criterion. A recognized state standardized exam was used as an outcome measure and its results were transparently reported. All quotes were checked against the source PDF and are verbatim. Criterion E is met because the study used California's SBAC state standardized ELA assessment as an outcome measure (in addition to the study-created writing assessment), and reported its results regardless of the null finding.
  • T

    Term Duration

    • Outcomes were measured in spring, roughly seven to eight months after the fall start of the year-long intervention, exceeding the one-term requirement.
    • "SRI administered a 2-day writing assessment to all students in focal classrooms in the fall as a baseline measure and again in spring as an outcome." (p. 38)
    • Relevant Quotes: 1) "The treatment is an intensive 46-hour PD program (via six full-day released days interspersed throughout the school year and five 2-hour after-school sessions)..." (p. 11) 2) "SRI administered a 2-day writing assessment to all students in focal classrooms in the fall as a baseline measure and again in spring as an outcome." (p. 38) 3) "Teachers administered the baseline prompt before learning of their randomization status and before the first PD session." (p. 39) 4) "In late April or early May, teachers administered the posttest." (p. 30) Detailed Analysis: The intervention began in the fall of the school year (the baseline writing assessment was administered in the fall, before the first PD session) and the outcome measurement (posttest) was administered in late April or early May, i.e., in spring of the same school year. The interval from intervention start to outcome measurement therefore spans roughly seven to eight months, covering the school year from fall to spring. This far exceeds the one-term (approximately 3-4 months) minimum tracking interval that criterion T requires. All quotes were checked against the source PDF and are verbatim. Criterion T is met because outcomes were measured about seven to eight months after the intervention began, well over one full academic term.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline equivalence, attrition, and business-as-usual treatment are documented in detail.
    • "Control teachers were asked to maintain business-as-usual conditions and were offered a 1-year version of the program at the end of the study." (p. 21)
    • Relevant Quotes: 1) "The teacher sample consisted of 230 secondary ELA and ELD teachers across grades 7-12, from 40 schools; 113 teachers were assigned to treatment and 117 to control." (p. 20) 2) "Students in both treatment and control conditions were roughly equivalent across most demographic characteristics (e.g., English learner status and race/ethnicity). (See Table A in the online supplementary materials)." (p. 20) 3) "Control teachers were asked to maintain business-as- usual conditions and were offered a 1-year version of the program at the end of the study." (p. 21) 4) "Control teachers conducted business as usual using the district English language arts textbook and core novels for teaching." (p. 31) 5) "This model predicted standardized baseline differences on the holistic AWC-LA of -.02 (p>.1), meeting the What Works Clearinghouse standards for baseline equivalence (IES, 2017)." (p. 41) 6) "The study's student attrition was 27% overall, 26% for treatment students, and 29% for control students." (p. 21) Detailed Analysis: The control group is well documented. The paper reports the number of control teachers (117) and their schools, demographic equivalence of control and treatment students (with a detailed supplementary table on English learner status and race/ethnicity), formally established baseline equivalence on the outcome measure (-.02 SD, meeting WWC standards), attrition rates by condition, and a clear description of what the control condition received (business-as-usual instruction with the district ELA textbook, plus the same half-day Collections textbook PD as treatment teachers, with the intervention offered only after the study). This documentation permits proper comparison and interpretation of results. All quotes were checked against the source PDF and are verbatim. Criterion D is met because the control group's size, demographics, baseline performance, and business-as-usual condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was at the teacher/class level within school-by-grade blocks, so whole schools were not the unit of assignment.
    • "Teachers were randomized together with the students in a single focal class, within school-by-grade blocks..." (p. 21)
    • Relevant Quotes: 1) "Teachers were randomized together with the students in a single focal class, within school-by-grade blocks..." (p. 21) 2) "Randomization occurred within grade-level blocks consisting of school-by-grade groupings, although not all schools had multiple study teachers in the same grade. In these cases, SRI formed blocks across schools, although teachers were still within the same grade and district." (pp. 21-22) 3) "The teacher sample consisted of 230 secondary ELA and ELD teachers across grades 7-12, from 40 schools..." (p. 20) Detailed Analysis: Criterion S requires that entire schools (or equivalent implementing institutions) be the unit of randomization. In this study, randomization occurred at the teacher/class level within schools: teachers within the same school and grade were split between treatment and control conditions (school-by-grade blocks explicitly contained both treatment and control teachers). The 40 participating schools therefore each could contain both conditions, and no quote indicates that whole schools were assigned to conditions. This is class-level, not school-level, randomization. All quotes were checked against the source PDF and are verbatim. Criterion S is not met because randomization was conducted at the teacher/class level within schools, not at the school level.
  • I

    Independent Conduct

    • SRI International, an independent nonprofit evaluator, performed randomization, data collection, and analysis with blinded scorers, ensuring independent conduct.
    • "Because of both the size and scope of the intervention, and requirement for an independent evaluator to collect and analyze all impact data, SRI International, an independent, nonprofit research center, conducted the intervention i3 evaluation using a cluster randomized controlled trial." (p. 13)
    • Relevant Quotes: 1) "Because of both the size and scope of the intervention, and requirement for an independent evaluator to collect and analyze all impact data, SRI International, an independent, nonprofit research center, conducted the intervention i3 evaluation using a cluster randomized controlled trial." (p. 13) 2) "SRI developed inclusion and exclusion criteria for NWP sites and local education agencies/districts, teachers, focal classes, and students..." (p. 19) 3) "Scorers did not know the students' treatment status or the time at which the paper was collected (pre- or posttest). Scorers (N=112) were recruited from current and former teachers affiliated with local NWP sites not participating in the intervention. SRI monitored the scoring to ensure the NWP followed impartial processes." (p. 39) 4) "SRI then randomized approximately half of the recruited teachers (and their classrooms) into treatment and half into control (delayed treatment)." (p. 21) Detailed Analysis: The intervention was developed and delivered by the UCI Writing Project team (the program developer) and partner NWP sites, and developer-affiliated authors appear on the paper. However, the i3 grant explicitly required an independent evaluator, and the paper documents that SRI International, an independent nonprofit research center, performed the randomization, collected and analyzed all impact data, defined the inclusion/exclusion criteria, and monitored scoring. In addition, the writing outcomes were scored by 112 raters from NWP sites not participating in the intervention who were blind to treatment status and test occasion. Following the ERCT examples for criterion I (e.g., evaluations led by a government or an external evaluation team while the developer is separately involved), explicit documentation that an external body independently collected and analyzed the outcome data satisfies the criterion even when developers co-author the paper. This assessment is corroborated by internet research: the 2024 SRI technical report on a later, separate Pathway evaluation (Woodworth, Arshan, & Wang, 2024) explicitly states "The SRI researchers who conducted this evaluation were external to and independent of UCIWP," confirming SRI's established role as an independent third-party evaluator of this intervention. All quotes were checked against the source PDF and are verbatim. Criterion I is met because an independent evaluator (SRI International) conducted the randomization, data collection, and impact analysis, with blinded external scorers, independent of the intervention developers.
  • Y

    Year Duration

    • The year-long intervention was tracked from fall to a late April/early May posttest, covering at least 75% of the academic year.
    • "The treatment is an intensive 46-hour PD program (via six full-day released days interspersed throughout the school year and five 2-hour after-school sessions)..." (p. 11)
    • Relevant Quotes: 1) "The treatment is an intensive 46-hour PD program (via six full-day released days interspersed throughout the school year and five 2-hour after-school sessions)..." (p. 11) 2) "Teachers administered the baseline prompt before learning of their randomization status and before the first PD session." (p. 39) 3) "SRI administered a 2-day writing assessment to all students in focal classrooms in the fall as a baseline measure and again in spring as an outcome." (p. 38) 4) "In late April or early May, teachers administered the posttest." (p. 30) 5) "Pathway is designed as a two-year intervention, where the first year introduces the program to teachers and the second year refines and reinforces their work with it. This paper reports only on Year 1 of the program..." (p. 12) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of an academic year after the intervention begins. The intervention (PD and classroom implementation) ran across the entire school year, beginning in fall after the baseline assessment, and the posttest was administered in late April or early May, near the end of the academic year. A US academic year runs roughly September to June (about 9-10 months); tracking from the fall start to late April/early May covers approximately 7.5-8 months, which is at least 75% of the academic year. The ERCT standard explicitly allows studies that are slightly shorter than a full academic year (by less than 25%) to satisfy this criterion. State SBAC scores collected at the spring administration likewise reflect essentially a full school year of exposure. All quotes were checked against the source PDF and are verbatim. Criterion Y is met because the intervention spanned the school year with outcomes measured in late spring, i.e., at least 75% of the academic year after the intervention began.
  • B

    Balanced Control Group

    • The extra teacher PD is the integral treatment variable tested against business-as-usual, and student instructional time was equal across conditions.
    • "In Year 1 85% of treatment teachers reported implementing the cognitive strategies tutorial and 92% reported using the revision tutorial; however, treatment and control teachers reported spending equal amounts of time teaching analytical essay writing and reading strategies." (p. 44)
    • Relevant Quotes: 1) "The treatment is an intensive 46-hour PD program (via six full-day released days interspersed throughout the school year and five 2-hour after-school sessions) in which secondary teachers learn how to integrate cognitive strategy instruction into process writing..." (p. 11) 2) "Control teachers were asked to maintain business-as- usual conditions and were offered a 1-year version of the program at the end of the study." (p. 21) 3) "Since Houghton Mifflin Harcourt donated class sets of the Collections textbook series as part of the i3 matching program, both treatment and control teachers attended a half day PD session devoted to introducing the Collections textbook." (p. 31) 4) "Treatment teachers reported receiving more ELA-focused PD (41 hours) than control teachers (9 hours)..." (p. 43) 5) "In Year 1 85% of treatment teachers reported implementing the cognitive strategies tutorial and 92% reported using the revision tutorial; however, treatment and control teachers reported spending equal amounts of time teaching analytical essay writing and reading strategies." (p. 44) Detailed Analysis: Applying the ERCT decision tree for criterion B: extra resources are present (treatment teachers received 46 hours of PD / 41 reported hours vs. 9 hours for control teachers). These extra resources are not a minor, negligible difference. However, the extra PD is not a separable add-on - it IS the intervention being tested: the study's explicit purpose is to determine whether the Pathway teacher PD program itself can be scaled up and improve student writing outcomes, so the additional PD time is the primary treatment variable (RESOURCES_ARE_ TREATMENT = true), and a business-as-usual PD control is appropriate by design (consistent with the ERCT criterion B examples, e.g., PAX-GBG and RAMSR, where PD/training that is integral to the treatment package does not violate the criterion). Critically, the resource differential is at the teacher-PD level, not the student-instructional-time level: survey data show treatment and control teachers spent equal amounts of time teaching analytical essay writing and reading strategies, so students themselves did not receive additional class time or budget. Both conditions also received the same half-day Collections textbook PD and textbook class sets, and control teachers were offered the full Pathway program after the study (delayed treatment/business-as-usual control). All quotes were checked against the source PDF and are verbatim. Criterion B is met because the extra teacher PD is the integral treatment variable being tested against a documented business-as-usual control, with no additional instructional time for treatment students.

• Level 3 Criteria

  • R

    Reproduced

    • All replications of the Pathway intervention, including a newer multi-state expansion trial, were run by the same developer/evaluator team (UCIWP/SRI) and none is published as an independent peer-reviewed replication.
    • "...the project applied for and received an Office of English Language Acquisition (OELA) grant to conduct an RCT to replicate the project in Anaheim Union High School District. This study also yielded significant and positive results (Olson et al. 2017...)." (p. 13)
    • Relevant Quotes: 1) "Developed by the UCI Writing Project, the Pathway Project began with an 8 year quasi-experimental longitudinal study in Santa Ana Unified School District (SAUSD)... Year 1 of that RCT (Kim et al., 2011) yielded an effect size of .35. Year 2 of the RCT yielded an effect size of .67 (Olson et al., 2012)." (pp. 12-13) 2) "...the project applied for and received an Office of English Language Acquisition (OELA) grant to conduct an RCT to replicate the project in Anaheim Union High School District. This study also yielded significant and positive results (Olson et al. 2017--Year 1, d=.48; Year 2, d=.60)." (p. 13) 3) "Further research is needed to determine whether the intervention can be successfully scaled up to a national level in more diverse geographical areas with a range of student populations and language backgrounds, and by implementers with less exposure to the National Writing Project." (p. 48) 4) "Our study does have limitations related both to the reasonably close physical proximity of the scale up sites to the original intervention site and the philosophical proximity of 'ideas and beliefs'... between the 'creators of the program to its first adopters and implementers'." (p. 48) External source (2024 SRI technical report, Woodworth, Arshan, & Wang): "This technical report describes a school-level randomized controlled trial of Pathway, implemented by UCIWP and seven expansion Writing Project sites in seven states (AZ, CA, MN, OK, TX, UT, and WI)... Despite disruptions to program implementation during the COVID-19 pandemic, the study found positive effects on student writing quality, conventions, and productivity... this evaluation - the second independent evaluation of Pathway when replicated by National Writing Project sites other than UCIWP - provides further evidence of Pathway's effectiveness." (Abstract, Conclusion, 2024) Detailed Analysis: Criterion R requires independent replication of the study by a different research team in a different context, published in a peer-reviewed journal. All prior and related Pathway trials (Olson & Land 2007, 2008; Kim et al. 2011; Olson et al. 2012; Olson et al. 2017) were conducted by the same developer team led by Carol Booth Olson at the UCI Writing Project; they are replications by the original authors, which the standard explicitly excludes. The present scale-up itself was delivered by closely affiliated NWP sites with the developer modeling every PD session, and the authors acknowledge the philosophical and geographic proximity as a limitation. Internet research located a subsequent 2018 federal EIR Expansion grant evaluation (Woodworth, Arshan, & Wang, 2023/2024) that tested Pathway in 46 schools across 7 states, delivered by 7 further expansion Writing Project sites trained by UCIWP. This is the closest candidate for an independent reproduction, but it does not satisfy criterion R for two reasons: (1) it was conducted by the same evaluation organization, SRI International, with two of the three authors (Katrina Woodworth and Nicole Arshan) being co-authors of the very paper under review here, so it is not an independent research team; and (2) as of this check, it has only been released as SRI technical reports/research briefs (2023, 2024), not as a peer- reviewed journal article, which the ERCT standard for criterion R requires. No other independently authored, peer-reviewed replication of this specific study (or its central experimental claim) by researchers unaffiliated with UCIWP/SRI was identified. All quotes were checked against the source PDF and are verbatim; the external quote was checked against the cited SRI technical report. Criterion R is not met because all replications of the Pathway intervention identified (including the newer EIR expansion evaluation) were conducted by the same developer- and-evaluator team, and none has been published as an independent, peer-reviewed replication by a different research team.
  • A

    All-subject Exams

    • Only ELA writing outcomes were assessed; no other main secondary school subjects were measured.
    • "The evaluation was a teacher-randomized controlled trial measuring the impact of the intervention on teachers' 7th- through 12th-grade English language arts (ELA) instructional practices and students' analytical writing achievement." (p. 18)
    • Relevant Quotes: 1) "SRI examined two types of student learning outcomes: one was writing data from study created prompts and the other were standardized assessments." (p. 38) 2) "Finally, SRI ran impacts on California's standardized ELA assessment, SBAC." (p. 42) 3) "The evaluation was a teacher-randomized controlled trial measuring the impact of the intervention on teachers' 7th- through 12th-grade English language arts (ELA) instructional practices and students' analytical writing achievement." (p. 18) Detailed Analysis: Criterion A requires standardized exam-based measurement of impact on all main subjects taught at the relevant school level, to detect possible negative side effects on non-target subjects. This study measured outcomes only in the English language arts domain: the on-demand analytical writing assessment and the SBAC ELA test. Secondary students in grades 7-12 study multiple core subjects (mathematics, science, social studies, etc.), none of which were assessed. No justification for a specialized exception (such as vocational training) applies here, since this is a general ELA intervention for mainstream secondary classrooms. All quotes were checked against the source PDF and are verbatim. Criterion A is not met because only ELA/writing outcomes were measured and no other main subjects (e.g., mathematics, science) were assessed.
  • G

    Graduation Tracking

    • Measurement stopped at the end of Year 1; students were not tracked through graduation, and no follow-up publication with graduation data was found.
    • "Pathway is designed as a two-year intervention, where the first year introduces the program to teachers and the second year refines and reinforces their work with it. This paper reports only on Year 1 of the program..." (p. 12)
    • Relevant Quotes: 1) "Pathway is designed as a two-year intervention, where the first year introduces the program to teachers and the second year refines and reinforces their work with it. This paper reports only on Year 1 of the program..." (p. 12) 2) "Ultimately, the long-term outcomes are increased English language arts test scores for all students, but especially high-need students, including ELs, and increased high-school graduation rates for those students..." (p. 18) 3) "In late April or early May, teachers administered the posttest." (p. 30) External source (2024 SRI technical report, Woodworth, Arshan, & Wang): "the UC Irvine Writing Project's (UCIWP's) Pathway to Academic Success program (Pathway) seeks to support students to meet their state-adopted ELA standards and graduate from high school prepared for college and work." (Abstract, 2024). The report's logic model lists "Distal Outcomes: Greater school success and persistence (graduation, college enrollment and persistence)" as an aspirational long-term outcome category, but reports no actual graduation, enrollment, or persistence data; its measured outcomes are limited to the AWC-LA writing assessment (quality, conventions, productivity) and state ELA/reading test scores, collected only through the end of each intervention year. Detailed Analysis: Criterion G requires that participants be tracked until graduation from their educational stage. This paper reports only Year 1 outcomes, with final measurement at the spring posttest and spring SBAC administration of the same school year. Although the project's logic model lists increased high-school graduation rates as an ultimate long-term outcome, no graduation data were collected or reported in this paper, and students in grades 7-12 were not followed to the end of secondary school. Internet research for follow-up publications by the same authors/team found the 2018 EIR Expansion grant evaluation of Pathway (Woodworth, Arshan, & Wang, 2023/2024), which similarly frames graduation as a long-term, logic-model aspiration but does not report any graduation-tracking data; its outcomes are limited to writing-assessment and state ELA test scores measured within each 1-2 year study window. No other follow-up study tracking this cohort (or subsequent Pathway cohorts) through to graduation was identified. All quotes were checked against the source PDF and are verbatim; the external quote was checked against the cited SRI technical report. Criterion G is not met because tracking ended with the Year 1 spring posttest (or, in the related EIR expansion study, with each intervention year's spring outcomes), and no graduation-tracking data have been published for this cohort in this paper or any identified follow-up.
  • P

    Pre-Registered

    • No public pre-registration of the study protocol (registry link, ID, or date) is mentioned in the paper or found via internet search; a later, separate Pathway study was registered, underscoring the absence of registration for this one.
    • Relevant Quotes: 1) "We gratefully acknowledge funding from the U.S. Department of Education's Investing in Innovation (i3) program, under grant number U411B130029." (p. 51) 2) "SRI International, an independent, nonprofit research center, conducted the intervention i3 evaluation using a cluster randomized controlled trial." (p. 13) Detailed Analysis: Criterion P requires that the full study protocol (hypotheses, methods, and planned analyses) be publicly pre-registered on a registry before data collection began, with the paper providing a registry reference and timing. The paper describes an i3 grant, an independent evaluator, confirmatory contrasts, and pre-determined fidelity thresholds, which suggest a structured evaluation plan, but it never mentions registration of the trial on any public registry (e.g., ClinicalTrials.gov, the Registry of Efficacy and Effectiveness Studies, or a published protocol) and provides no registration ID or date that would allow verifying that registration preceded data collection. Internet research confirmed this study (the 2014 i3 Validation grant evaluation, underlying this 2020 paper) has no located pre-registration entry. By contrast, a later, separate Pathway evaluation funded by a 2018 EIR Expansion grant (Woodworth, Arshan, & Wang, 2024) was explicitly preregistered by SRI in the IES-funded REES registry (Study Registry ID 3520) on May 28, 2021, with a documented amendment on September 2, 2022 - this demonstrates the research team's general practice of registering studies once required by later grant terms, but it is a distinct study (different grant, different sites, different years) from the paper under review, and confirms by contrast that no equivalent registration exists for this 2014-2018 i3 Validation study. No pre-registration of this specific study's protocol was found in any source. Criterion P is not met because no reference to, or record of, a public pre-registration of this study's protocol (registry link, ID, or date) was found in the paper or via independent internet search.