Learning Strategy Applications with Students of English as a Second Language

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation in the Phase II training study was conducted at the individual student level within each school, not at the class or school level, and the intervention was group instruction rather than one-to-one tutoring, so no exception applies.
    • "Students were randomly assigned to one of three instructional groups in each school." (p. 570)
    • Relevant Quotes: 1) "Students were randomly assigned to one of three instructional groups in each school." (p. 570) 2) "Each school's groups were roughly proportional to ethnic and sex ratios within that school." (p. 570) 3) "The size of the instructional group within each school averaged eight to ten students." (p. 570) Detailed Analysis: The paper explicitly describes random assignment of individual students, drawn from the ESL population within each of the three participating high schools, into instructional groups of eight to ten students each (metacognitive, cognitive, control). This is individual/student-level randomisation, not class-level or school-level randomisation, and it creates a risk that treatment and control students from the same school (and possibly the same ESL classes) could interact and contaminate each other. The intervention (group-based strategy instruction delivered by project staff) is not personal one-to-one tutoring, so the tutoring exception described in the ERCT standard does not apply. Criterion C is not met because randomisation occurred at the individual student level within schools rather than at the class or school level, and no valid exception applies.
  • E

    Exam-based Assessment

    • Outcomes were measured with custom, researcher-designed listening and speaking instruments rather than widely recognised standardised exams.
    • "A short listening comprehension test following each lecture contained items designed to assess Bloom and Krathwohl's (1977) knowledge, comprehension, and analysis levels." (p. 571)
    • Relevant Quotes: 1) "The videotapes were designed to simulate a lecture experience the students might encounter in school." (p. 571) 2) "A short listening comprehension test following each lecture contained items designed to assess Bloom and Krathwohl's (1977) knowledge, comprehension, and analysis levels... All pretests, post-tests, and interim assessments were multiple-choice recognition items." (p. 571) 3) "Taped oral presentations at pretest and post-test were scored blind by five judges, who rated the speeches on a 1-5 scale reflecting delivery (volume and pace), appropriateness (choice of words and phrases for a class presentation), accuracy (phonological, syntactic, and semantic), and organization (coherence and cohesion)." (p. 573) Detailed Analysis: Neither outcome measure is a standardised, widely recognised exam. The listening test was a purpose-built, multiple-choice instrument tied to researcher-produced videotaped lectures on specific academic topics, and the speaking measure was a researcher-devised 1-5 rating scale scored by five recruited judges, not any established, externally validated language-proficiency test. Both instruments were created specifically for this study and its training content rather than adopted from an existing standardised assessment. Criterion E is not met because the study used custom-built listening and speaking measures rather than standardised, widely recognised exams.
  • T

    Term Duration

    • The entire intervention and outcome measurement spanned only about two weeks, far short of the one-term minimum.
    • "Students received instruction and practice in the use of learning strategies for 50 minutes daily for 8 days." (p. 572)
    • Relevant Quotes: 1) "Students received instruction and practice in the use of learning strategies for 50 minutes daily for 8 days." (p. 572) 2) "In addition, a full 50-minute period was used for pretesting and another for post-testing." (p. 572) 3) "Each student made four separate oral presentations on four separate days." (pp. 572-573) Detailed Analysis: From pretest through post-test, the Phase II training study spanned roughly 8 instructional days plus one pretest session and one post-test session, i.e. about two school weeks in total. There is no indication anywhere in the paper that outcomes were measured any later than immediately following the final training day. This is far shorter than the one full academic term (roughly 3-4 months) required by the ERCT standard. Criterion T is not met because the intervention and its outcome measurement together covered only about two weeks.
  • D

    Documented Control Group

    • The control group's size, composition, and the specific business-as-usual activities it received instead of strategy training are clearly documented.
    • "The time used for learning strategy instruction for the experimental groups was used for reading comprehension activities with the control group so that they too would benefit from extra instruction (even though reading was not investigated in this study)." (p. 572)
    • Relevant Quotes: 1) "A random control group received the same tasks but with no strategy training." (p. 562) 2) "Control (n = 22)" and adjusted means for the control group on listening and speaking post-tests (Table 3, p. 575) 3) "Students in the control group received no strategy instruction. They were simply told to listen to the videotapes and do whatever they normally did to help them understand and remember a lecture... The time used for learning strategy instruction for the experimental groups was used for reading comprehension activities with the control group..." (p. 572) 4) "The control group, which received no strategy instruction, was given the list of topic possibilities and told to prepare an oral report on the topic of their choice in whatever manner they normally prepared for such an activity... students were not instructed to provide systematic feedback to their peers." (pp. 573-574) Detailed Analysis: The paper documents the control group's exact size (n=22), notes that group composition was "roughly proportional to ethnic and sex ratios" within each school (drawn from the same 75- student intermediate-level ESL sample), and describes precisely what the control condition did instead of strategy training for both the listening task (ordinary listening plus substitute reading-comprehension activities) and the speaking task (self-directed report preparation without structured peer feedback). Pretest scores were also collected and used as a covariate, allowing baseline comparability to be checked statistically. Criterion D is met because the control group's size, demographic composition, and specific conditions are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the individual student level within each school rather than between schools.
    • "Students were randomly assigned to one of three instructional groups in each school." (p. 570)
    • Relevant Quotes: 1) "Students were randomly assigned to one of three instructional groups in each school." (p. 570) 2) "The subjects were 75 high school students enrolled in ESL classes during the Fall 1983 semester in three Eastern suburban high schools." (p. 570) Detailed Analysis: All three participating schools contributed students to all three conditions (metacognitive, cognitive, control); schools themselves were not the unit of random assignment, and no school was assigned wholesale to a single condition. Students within each school were individually randomised into groups. This is a weaker design than school-level randomisation and does not satisfy the stronger S criterion. Criterion S is not met because randomisation occurred within, rather than between, schools.
  • I

    Independent Conduct

    • The same research team that designed the strategy taxonomy and training curriculum also delivered the intervention and analysed the results, with no independent third-party conduct described.
    • "To control for teacher effects, three project staff alternated presenting the three treatment conditions at the three participating schools." (p. 570)
    • Relevant Quotes: 1) "J. Michael O'Malley is a Senior Associate at InterAmerica Research Associates... He was the Principal Investigator for this study..." (p. 578, The Authors) 2) "Anna Uhl Chamot... was the ESL Specialist for this study..." (p. 578, The Authors) 3) "To control for teacher effects, three project staff alternated presenting the three treatment conditions at the three participating schools." (p. 570) 4) "This research was conducted for the U.S. Army Research Institute for the Behavioral and Social Sciences in Alexandria, Virginia, under Contract No. MDA 903-82-C-0169." (p. 578, Acknowledgments) Detailed Analysis: The InterAmerica Research Associates team that developed the metacognitive/cognitive/ socioaffective strategy taxonomy and designed the training materials (drawing on their own 1983 literature review) also personally delivered the training ("three project staff alternated presenting the three treatment conditions") and analysed the outcome data. Funding came from a U.S. Army research contract, but no independent, third-party organisation is described as having conducted data collection, delivered the intervention, or analysed results independently of the intervention's designers. Criterion I is not met because the intervention designers themselves delivered the training and conducted the evaluation.
  • Y

    Year Duration

    • Because criterion T (Term Duration) is not met, criterion Y is automatically not met; independently, the actual duration was only about two weeks.
    • "Students received instruction and practice in the use of learning strategies for 50 minutes daily for 8 days." (p. 572)
    • Relevant Quotes: 1) "Students received instruction and practice in the use of learning strategies for 50 minutes daily for 8 days." (p. 572) 2) "In addition, a full 50-minute period was used for pretesting and another for post-testing." (p. 572) Detailed Analysis: Per the standard's cascading rule, since criterion T is not met, criterion Y cannot be met either. Independently, the training study ran for only about two weeks from pretest to post-test, which is nowhere near the 75% of an academic year (roughly 9-10 months) required for Y. Criterion Y is not met both because criterion T is not met and because the study's actual duration was only about two weeks.
  • B

    Balanced Control Group

    • Instructional time was explicitly equalised across the metacognitive, cognitive, and control groups, with the manipulated variable being the strategy instruction itself.
    • "The time used for learning strategy instruction for the experimental groups was used for reading comprehension activities with the control group so that they too would benefit from extra instruction (even though reading was not investigated in this study)." (p. 572)
    • Relevant Quotes: 1) "Report preparation was completed in class to ensure comparable time on task across treatment groups." (p. 573) 2) "Students in the control group received no strategy instruction... The time used for learning strategy instruction for the experimental groups was used for reading comprehension activities with the control group so that they too would benefit from extra instruction (even though reading was not investigated in this study)." (p. 572) 3) "The control group... was given the list of topic possibilities and told to prepare an oral report on the topic of their choice... This group also tape-recorded reports in the presence of a small group of students during practice sessions." (pp. 573-574) Detailed Analysis: The additional resource under investigation here is the strategy training itself (the treatment variable), not extra class time or budget. All three groups received the same number of sessions, the same session length, the same videotape lectures and speaking topics, and the same pretest/post-test schedule. For the listening task, the control group's instructional period was explicitly filled with reading comprehension activities rather than left empty, deliberately equalising instructional time across conditions. For the speaking task, the control group prepared and tape-recorded reports on the same schedule and with "comparable time on task"; the only substantive difference was the presence or absence of strategy instruction and structured peer feedback, which is exactly the variable being tested. Applying the decision procedure: no extra time or budget was given to the intervention groups beyond the strategy content itself, so the criterion is met at the first branch (no extra resources present). Criterion B is met because instructional time was explicitly equalised across groups via matched activities and time-on-task, and the only difference was the strategy-training variable itself.

• Level 3 Criteria

  • R

    Reproduced

    • No evidence was found of this specific training experiment having been independently replicated by a different research team in a peer-reviewed outlet.
    • Relevant Information: This paper's Phase I descriptive findings were also reported by the same author team in a companion article, "Learning strategies used by beginning and intermediate ESL students" (O'Malley, Chamot, Stewner-Manzanares, Küpper, and Russo 1985, Language Learning 35(1):21-46), but this is by the same research group and covers the same descriptive data, not an independent replication of the Phase II training experiment. Detailed Analysis: Targeted internet searches (general web search and academic literature search) for independent replications of this specific Phase II training study (metacognitive/cognitive/socioaffective strategy groups versus a control group on listening and speaking outcomes among intermediate-level high school ESL students) were conducted. Citing literature discussing this 1985 study (e.g., reviews of learning strategy training research) was located, but no independent research team's peer-reviewed replication of this specific study design and sample was identified. Criterion R is not met because no independent replication of this specific study was found after internet-backed searching.
  • A

    All-subject Exams

    • Because criterion E (Exam-based Assessment) is not met, criterion A is automatically not met; the study also assessed only English listening and speaking, not other subjects.
    • Detailed Analysis: Per the standard's prerequisite rule, since criterion E is not met (the outcome measures were custom-built rather than standardised), criterion A cannot be met either. Independently, the study's outcome measures (listening comprehension test scores and speaking ratings) cover only English-language skills and do not assess performance in any other main subject area. Criterion A is not met because criterion E is not met and only English listening/speaking outcomes were assessed.
  • G

    Graduation Tracking

    • Because criterion Y (Year Duration) is not met, criterion G is automatically not met; no follow-up beyond the immediate post-test is described, and no subsequent tracking papers were located.
    • Detailed Analysis: Per the standard's cascading rule, since criterion Y is not met, criterion G cannot be met either. Independently, the paper describes only an 8-day training period bracketed by one pretest and one post-test session, with no mention of any follow-up tracking of students after the post-test, let alone through graduation. Internet searches for subsequent publications by the same author team (O'Malley, Chamot, Stewner-Manzanares, Russo, Küpper) tracking this Fall 1983 Phase II training cohort toward graduation were conducted. No such follow-up publication was found; the only related paper identified is the companion Phase I descriptive article in Language Learning (1985), which does not track the Phase II training sample toward graduation. Criterion G is not met because criterion Y is not met and no post-intervention or graduation follow-up publication was found.
  • P

    Pre-Registered

    • No pre-registration of the study's hypotheses, methods, or analysis plans is mentioned anywhere in the paper, and none would be expected given the study predates modern trial registries.
    • Relevant Quotes: 1) "This research was conducted for the U.S. Army Research Institute for the Behavioral and Social Sciences in Alexandria, Virginia, under Contract No. MDA 903-82-C-0169." (p. 578, Acknowledgments) Detailed Analysis: No statement referencing a public registry, registration date, or pre-specified analysis plan appears anywhere in the paper. As a 1985 study, it predates modern clinical/educational trial registries (e.g., ClinicalTrials.gov, AsPredicted, OSF Registries), which were not established until decades later. Internet searches for any registry record referencing this study or its authors' 1983 contract found no pre-registration evidence. Criterion P is not met because no pre-registration reference or date is provided, and no registry record was found.