Reducing Achievement Gaps in Academic Writing for Latinos and English Learners in Grades 7-12

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Teachers (and their single focal class) rather than individual students within one class were randomly assigned to condition, satisfying class-level randomization.
    • "Ninety-five teachers in 16 secondary schools were stratified by school and grade and then randomly assigned to the Pathway or control group."
    • Relevant Quotes: 1) "This study utilized a multisite, cluster, randomized field trial in which secondary schools were the sites, teachers were randomly assigned to the Pathway Project (hereafter, 'Pathway' condition or group) or control condition in the intervention, and one of each teacher's classes was randomly selected to participate in the study." (p. 5) 2) "Ninety-five teachers in 16 secondary schools were stratified by school and grade and then randomly assigned to the Pathway or control group." (abstract, p. 1) 3) "In the first step of a three-step process, school counselors employed a software program to randomly assign students to teachers' classrooms. Next, recruited teachers were randomly assigned to either Pathway or control conditions." (p. 5) Detailed Analysis: All three quotes were checked against the PDF and are verbatim. The unit of random assignment to condition is the teacher (and, by extension, that teacher's single selected focal class), not individual students within one classroom. Teachers were stratified by school and grade and then randomized, and only one class per teacher participated, so no classroom contained both treatment and control students. This satisfies the requirement that randomization occur at the class level or higher, avoiding within-classroom contamination, even though it falls short of a full school-level RCT (see criterion S) because individual schools contained both Pathway and control teachers. Criterion C is met because randomization occurred at the teacher/class level, not within a single classroom.
  • E

    Exam-based Assessment

    • In addition to a custom writing measure, the study reports outcomes on the CAHSEE, a recognized standardized state exam, satisfying the standardized-assessment requirement.
    • "The CAHSEE is a standardized literacy assessment that is aligned with the California English/Language Arts academic content standards through Grade 10 and is a graduation requirement for all high school students."
    • Relevant Quotes: 1) "The AWA is a measure developed for the Pathway Project to assess the analytical writing skills of secondary school students." (p. 11) 2) "outcome measures including pre- and posttest on-demand AWAs administered by the participant teachers and the California High School Exit Exam (CAHSEE)." (p. 10) 3) "The CAHSEE is a standardized literacy assessment that is aligned with the California English/Language Arts academic content standards through Grade 10 and is a graduation requirement for all high school students." (p. 11) Detailed Analysis: All three quotes were checked against the PDF and are verbatim. The primary, program-specific outcome measure (the AWA) is a study-created, on-demand writing assessment developed specifically for the Pathway Project and is therefore not a standardized exam on its own. However, the study also reports impacts on the CAHSEE, which is explicitly described as a standardized, state-mandated exam aligned to state ELA content standards and used as a graduation requirement. Because the paper reports outcomes on this widely recognized standardized exam in addition to the custom AWA, the requirement for use of a standard exam-based assessment is satisfied. Criterion E is met because the study measured impact using the CAHSEE, a recognized, standardized exam, alongside the custom AWA.
  • T

    Term Duration

    • Measurement (CAHSEE in March, AWA posttest and classroom observations through Spring) occurred well beyond a single term after the October pretest/intervention start.
    • "the CAHSEE was administered to all 10th graders during 2013 and 2014. Tenth graders first sit for this exam during March of each year."
    • Relevant Quotes: 1) "Data collection took place from September 2012 until June of 2014." (p. 10) 2) "In October of each year, students in the classes of Pathway and control teachers wrote a timed, on-demand, text-based academic essay interpreting a theme ... to determine their growth as academic writers over time." (p. 11) 3) "the CAHSEE was administered to all 10th graders during 2013 and 2014. Tenth graders first sit for this exam during March of each year." (p. 11) Detailed Analysis: All three quotes were checked against the PDF and are verbatim (quote 2 elides the parenthetical list of essay prompts between "theme" and "to determine"). Pretest writing assessments occurred in October of each program year, with the CAHSEE administered the following March and further outcome collection (posttest AWA, classroom observations in Winter and Spring) continuing through the end of each school year. This interval clearly exceeds a single academic term (3-4 months). Additionally, because the stronger Year Duration criterion (Y, below) is met, the ERCT standard specifies that this weaker Term Duration criterion is automatically considered met as well. Criterion T is met because outcome measurement occurred well beyond one academic term after the intervention began, and because criterion Y is also met.
  • D

    Documented Control Group

    • The control group's demographics, baseline scores, and business-as-usual instructional conditions are documented in detail in the text and in Tables 1-2.
    • "In contrast, control teachers conducted business as usual, using the district English language arts textbook and core novels for teaching. Both groups attended one full day of professional development led by district curriculum specialists on protocols for reviewing district benchmark assessments."
    • Relevant Quotes: 1) "In contrast, control teachers conducted business as usual, using the district English language arts textbook and core novels for teaching. Both groups attended one full day of professional development led by district curriculum specialists on protocols for reviewing district benchmark assessments." (p. 10) 2) "Table 1 shows descriptive statistics for Pathway and control group students in the first year of the project, as well as a test of the significance of between-groups differences." (p. 5) 3) "There was no statistically significant difference between Pathway teachers and control teachers in the total years of teaching experience (p = .45) ... These findings indicate that Pathway and comparison group teachers were similar on observed teacher characteristics measured at baseline." (p. 5) 4) "46 teachers, 1,493 students" / "49 teachers, 1,705 students" (Table 1 notes, p. 5) Detailed Analysis: All four quotes were checked against the PDF and are verbatim. The control condition is described explicitly ("business as usual," district textbook and core novels, one shared day of district-led PD), and Tables 1 and 2 provide detailed baseline demographic and achievement statistics (grade, gender, race/ethnicity, EL/RFEP/IFEP/EO status, FRPL eligibility, pretest AWA and CST scores) for the control group in both years, along with significance tests versus the Pathway group. Criterion D is met because the control group's size, demographics, baseline performance, and business-as-usual condition are documented in detail.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the teacher level within each school (stratified by school and grade), not by randomizing whole schools to a single condition.
    • "Ninety-five teachers in 16 secondary schools were stratified by school and grade and then randomly assigned to the Pathway or control group."
    • Relevant Quotes: 1) "Ninety-five teachers in 16 secondary schools were stratified by school and grade and then randomly assigned to the Pathway or control group." (abstract, p. 1) 2) "This study utilized a multisite, cluster, randomized field trial in which secondary schools were the sites, teachers were randomly assigned to the Pathway Project ... or control condition." (p. 5) Detailed Analysis: Randomization was carried out at the teacher level, stratified within each school and grade, meaning that each of the 16 participating secondary schools contained both Pathway teachers and control teachers side by side. Whole schools were not assigned wholesale to a single condition; schools served merely as the "sites" from which teachers were drawn and stratified. Because entire schools were not randomized to one condition or the other, this does not meet the stronger school-level RCT requirement. Criterion S is not met because randomization occurred within schools at the teacher level, not between whole schools.
  • I

    Independent Conduct

    • The intervention's own developers led the teacher training and the same research team conducted the study and analysis, with no independent evaluator described.
    • "Training was led by the developers of the Pathway Project and supported by literacy coaches who participated as Pathway teachers in a previous quasi-experimental research study."
    • Relevant Quotes: 1) "Training was led by the developers of the Pathway Project and supported by literacy coaches who participated as Pathway teachers in a previous quasi-experimental research study (Olson & Land, 2007)." (p. 6) 2) "Correspondence concerning this article should be addressed to Carol Booth Olson, Director of the UCI Writing Project, School of Education, University of California, Irvine." (author note, p. 1) 3) "The Pathway Project was initially developed over an 8-year period in the SAUSD (Olson & Land, 2007), in response to the specific needs of teachers serving a rapidly increasing population of ELs." (p. 6) Detailed Analysis: All three quotes were checked against the PDF and are verbatim. The Pathway Project intervention was designed and developed by Carol Booth Olson (a co-author and Director of the UCI Writing Project) together with the same research group, and the professional development itself was "led by the developers of the Pathway Project." These same authors also carried out the study design, teacher training, data analysis, and reporting of results. Nowhere in the Method, Data Sources, or Data Analytic Strategy sections is there mention of an external, third-party organization independently implementing or evaluating the intervention; the research team that created the intervention is the same team that conducted and analyzed the trial. Criterion I is not met because the intervention's developers led the training and the same author team conducted the evaluation and analysis, with no stated independent third-party oversight.
  • Y

    Year Duration

    • Training, coaching, and outcome measurement (pretest in October through Spring observations and March CAHSEE) spanned nearly the entire academic year in each of the two program years.
    • "Pathway teachers in the intervention participated in 46 hr of training each school year (via six 6-hr released days interspersed throughout the school year and five 2-hr after-school sessions)."
    • Relevant Quotes: 1) "Data collection took place from September 2012 until June of 2014." (p. 10) 2) "Pathway teachers in the intervention participated in 46 hr of training each school year (via six 6-hr released days interspersed throughout the school year and five 2-hr after-school sessions)." (p. 6) 3) "During Year 1, we observed 91 of the participating teachers (Pathway and control) for one class period (46 min) at two time points (Winter 2012 and Spring 2013)." (p. 11) 4) "In October of each year, students in the classes of Pathway and control teachers wrote a timed, on-demand, text-based academic essay ... to determine their growth as academic writers over time." (p. 11) Detailed Analysis: All four quotes were checked against the PDF and are verbatim. Each program year runs from an October pretest through training delivered across the whole school year (six released days plus five after-school sessions "interspersed throughout the school year"), coaching observations at Winter and Spring time points, and a March CAHSEE administration, with the overall data collection window spanning September 2012 to June 2014 across the two program years. This design tracks students essentially across each full academic year (well over 75% of an academic year, ~9-10 months), rather than a brief, short-term intervention window. Criterion Y is met because the intervention and outcome measurement span nearly the full academic year in each of the two program years.
  • B

    Balanced Control Group

    • The extra teacher training/coaching time is the explicit treatment variable being tested (a PD program), so the control group's business-as-usual, non-Pathway PD is an appropriate baseline rather than an unbalanced omission.
    • "This study reports 2 years of findings from a randomized controlled trial designed to replicate and demonstrate the efficacy of an existing, successful professional development program, the Pathway Project."
    • Relevant Quotes: 1) "This study reports 2 years of findings from a randomized controlled trial designed to replicate and demonstrate the efficacy of an existing, successful professional development program, the Pathway Project." (abstract, p. 1) 2) "Pathway teachers in the intervention participated in 46 hr of training each school year (via six 6-hr released days interspersed throughout the school year and five 2-hr after-school sessions)." (p. 6) 3) "In contrast, control teachers conducted business as usual, using the district English language arts textbook and core novels for teaching. Both groups attended one full day of professional development led by district curriculum specialists on protocols for reviewing district benchmark assessments." (p. 10) Detailed Analysis: All three quotes were checked against the PDF and are verbatim. Pathway teachers received substantially more professional development time (46 hr/year plus ongoing coaching) than control teachers, who received only one shared day of district-led PD on benchmark assessments. Re-applying the current ERCT criterion B decision logic: extra resources ARE present (46 hr vs. 1 day of PD), the difference is not negligible, so the key question is whether these extra resources are integral to the design (the treatment being tested) rather than a separable add-on. The study's explicit purpose is to test "the efficacy of an existing ... professional development program" -- i.e., the additional teacher training and coaching time is itself the treatment variable under investigation, not a supplementary, separable resource layered on top of some other core intervention. Matching the training hours in the control condition would directly confound the very question the study is designed to answer, so the RESOURCES_ARE_TREATMENT branch of the decision tree applies and the control's "business as usual" condition is the appropriate baseline. Consistent with this, both groups received the same one day of non-Pathway, district-standard PD, and control classrooms otherwise continued ordinary curriculum and materials. Criterion B is met because the additional professional development hours are integral to, and are precisely, the treatment variable being evaluated, and the control group appropriately received a business-as-usual level of support.

• Level 3 Criteria

  • R

    Reproduced

    • The replication described is of the same intervention by overlapping authors/developers (the Pathway Project team), and internet search found no genuinely independent replication, so the independent-reproduction requirement is not satisfied.
    • "Our main finding is that our randomized controlled trial of the Pathway Project in the AUHSD largely replicated the earlier RCT in the SAUSD."
    • Relevant Quotes: 1) "This study reports 2 years of findings from a randomized controlled trial designed to replicate and demonstrate the efficacy of an existing, successful professional development program, the Pathway Project." (abstract, p. 1) 2) "Our main finding is that our randomized controlled trial of the Pathway Project in the AUHSD largely replicated the earlier RCT in the SAUSD (Year 1 SAUSD, d = .35, and Year 2, SAUSD d = .67; Year 1 AUHSD, d = .48, and Year 2, d = .60)." (p. 16) 3) "This logic model was developed based on findings from an earlier 4-year randomized controlled trial conducted in the Santa Ana Unified School District (SAUSD) ... (Kim et al., 2011; Olson et al., 2012)." (p. 4) Internet Research (2026-07-30): A search of the ~70 works citing this paper (via OpenAlex, work ID W2344291594) and general web searches did not surface any independent replication of this specific AUHSD study, or of the broader Pathway Project intervention model, conducted by a research team unrelated to Carol Booth Olson and her collaborators. The only subsequent, closely related study found is: Olson, C. B., Woodworth, K., Arshan, N., Black, R., Chung, H. Q., D'Aoust, C., Dewar, T., Friedrich, L., Godfrey, L., Land, R., Matuchniak, T., Scarcella, R., & Stowell, L. (2020). "The pathway to academic success: Scaling up a text-based analytical writing intervention for Latinos and English learners in secondary school." Journal of Educational Psychology, 112(4), 701-717. https://doi.org/10.1037/edu0000387. This 2020 study again lists Carol Booth Olson (the intervention's developer) as corresponding/lead author, with several other overlapping Pathway Project personnel (e.g., Huy Q. Chung, Tina Matuchniak, Robert Land, Robin Scarcella), so it is a scale-up by the same developer team rather than an independent reproduction. No quotes from a genuinely independent replication could be located, and none are fabricated here. Detailed Analysis: All three in-paper quotes were checked against the PDF and are verbatim. The paper explicitly presents itself as a replication of an earlier RCT of the same Pathway Project intervention conducted in the Santa Ana Unified School District. However, that earlier trial (Olson & Land, 2007; Kim et al., 2011; Olson et al., 2012) shares overlapping authors with the current study -- notably Carol Booth Olson, who both developed the intervention and co-authored the prior SAUSD evaluations. The ERCT standard's Reproduced criterion requires replication by a genuinely independent research team (a different team or institution), not the same developers/authors replicating their own earlier work in a new district or, as the 2020 scale-up study shows, with a larger sample. No evidence was found, either in the text or via internet search, that an unrelated, external research group has independently replicated this specific AUHSD study or the Pathway Project model more broadly. Criterion R is not met because the "replication" referenced in the paper, and the only subsequent related study located via internet search, were conducted by the same research team/developers as the original study, not by an independent research team.
  • A

    All-subject Exams

    • Only English Language Arts/writing outcomes (AWA, CAHSEE) were measured; no other core subjects were assessed.
    • "outcome measures including pre- and posttest on-demand AWAs administered by the participant teachers and the California High School Exit Exam (CAHSEE)."
    • Relevant Quotes: 1) "outcome measures including pre- and posttest on-demand AWAs administered by the participant teachers and the California High School Exit Exam (CAHSEE)." (p. 10) 2) "The CAHSEE is a standardized literacy assessment that is aligned with the California English/Language Arts academic content standards through Grade 10." (p. 11) Detailed Analysis: Both quotes were checked against the PDF and are verbatim. All reported outcome measures (the custom AWA and the CAHSEE) assess English Language Arts / writing skills only. No outcomes are reported for other core subjects such as mathematics or science, and the paper offers no stated rationale for restricting assessment to a single subject area (the exception allowed for highly specialized vocational interventions does not apply here, since this is a general secondary ELA writing intervention). Because criterion A additionally requires that criterion E be fully met with standardized, all-subject coverage, and only ELA-related standardized outcomes are reported, the all-subject requirement is not satisfied. Criterion A is not met because only English Language Arts/writing outcomes were assessed, with no measurement of other core subjects.
  • G

    Graduation Tracking

    • Outcomes were tracked only within each one-year program cycle; the identified follow-up (2020 scaling-up study) expands to new students rather than tracking this cohort, and no evidence of follow-up through graduation was found.
    • "This study reports 2 years of findings from a randomized controlled trial."
    • Relevant Quotes: 1) "Data collection took place from September 2012 until June of 2014." (p. 10) 2) "This study reports 2 years of findings from a randomized controlled trial designed to replicate and demonstrate the efficacy of an existing, successful professional development program, the Pathway Project ..." (abstract, p. 1) 3) "In a current study involving over 8,000 students from four large, urban school districts in the Southern California region, we hope to compare the impact of the Pathway Project on an even more diverse student population." (p. 17) Internet Research (2026-07-30): Following up on quote 3, the "current study" referenced in the paper's limitations section was identified as: Olson, C. B., Woodworth, K., Arshan, N., Black, R., Chung, H. Q., D'Aoust, C., Dewar, T., Friedrich, L., Godfrey, L., Land, R., Matuchniak, T., Scarcella, R., & Stowell, L. (2020). "The pathway to academic success: Scaling up a text-based analytical writing intervention for Latinos and English learners in secondary school." Journal of Educational Psychology, 112(4), 701-717. https://doi.org/10.1037/edu0000387 (open-access copy at https://escholarship.org/content/qt4b14h6j4/qt4b14h6j4.pdf). This 2020 paper is a scale-up implementation study with a new, much larger sample (~8,000 students across four additional school districts and 230 teachers); it is not a longitudinal follow-up of the original 2012-2014 AUHSD student cohort, and it does not report tracking those specific students through graduation. No other follow-up publication tracking the AUHSD cohort to graduation was found via a citation search of the ~70 works citing this paper (OpenAlex work ID W2344291594) or via general web search. I could not find evidence of graduation tracking and have not fabricated any quotes to suggest otherwise. Detailed Analysis: All three quotes were checked against the PDF and are verbatim (quote 2 is truncated at a natural point mid sentence, marked with an ellipsis). The study follows two overlapping cohorts of students (Grades 7-12) for one program year each, across two school years total (2012-2013 and 2013-2014), with outcome data collected via pretest/posttest AWA scores and 10th-grade CAHSEE pass rates within each year. There is no report of continued tracking of any student cohort beyond the program year in which they were enrolled, and certainly no tracking through to high school graduation. The "current study" mentioned in the limitations section, confirmed via internet research to be the 2020 scale-up paper, describes a planned expansion to more districts and students rather than an extension of follow-up duration for the cohorts in this study. Criterion G is not met because there is no evidence, in this paper or in subsequent publications located via internet search, of follow-up tracking of participants through graduation.
  • P

    Pre-Registered

    • No mention of pre-registration, a registry ID, or registration date is found anywhere in the paper, and no registry record was found via internet search.
    • Relevant Quotes: No quotes referencing pre-registration, a registry platform, a registration ID, or a registration date were found anywhere in the Method, Data Analytic Strategy, or other sections of the paper. Internet Research (2026-07-30): Searches of general web sources, the Open Science Framework (OSF), and the ERIC record for this paper (ERIC number EJ1125530) did not surface any pre-registration record for this study's protocol, hypotheses, or analysis plan (e.g., on OSF, an IES/SREE effectiveness registry, or a clinical trials-style registry), nor for the federal OELA grant (T365Z120153) that funded it. No pre-registration quotes could be located, and none have been fabricated here. Detailed Analysis: The manuscript contains no statement that the study protocol, hypotheses, or planned analyses were registered on a public registry (e.g., a clinical-trials-style or education-trial registry) prior to data collection. Given the complete absence of any such reference in the paper and the absence of any registry record found via internet search, this criterion cannot be considered met. Criterion P is not met because no pre-registration statement, registry link, or registration date appears anywhere in the paper or in available registries.