Effect of cooperative learning on the acquisition of second language rules and mechanics

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the level of whole intact classes, satisfying the class-level RCT requirement.
    • "The 12 intact classes were randomly assigned to 6 control and 6 experimental conditions." (p. 225)
    • Relevant Quotes: 1) "Four fourth-grade, four fifth-grade, and four sixth-grade intact classes (n = 318 students) were randomly assigned to experimental and control conditions." (p. 223, Abstract) 2) "The 12 intact classes were randomly assigned to 6 control and 6 experimental conditions." (p. 225) 3) "That is, 4 fourth-grade classes (2 control and 2 experimental; n = 115 students), 4 fifth-grade classes (2 control and 2 experimental; n = 94 students), and 4 sixth-grade classes (2 control and 2 experimental; n = 109 students) participated in the study." (p. 225) Detailed Analysis: The unit of randomisation was clearly the intact class, not the individual student within a class. Twelve whole classes were assigned as complete units to either the control or experimental condition, which prevents contamination between treatment and control students who sit in the same room. This satisfies the class-level RCT requirement. Final sentence: Criterion C is met because randomisation was performed at the level of entire intact classes rather than individual students.
  • E

    Exam-based Assessment

    • The study used custom, researcher-developed domain-referenced tests rather than a widely recognised standardised exam.
    • "The pre-tests were specifically developed for the purpose of the present study..." (p. 225)
    • Relevant Quotes: 1) "The pre-tests were specifically developed for the purpose of the present study and were used to control for any initial differences in the performance of the control and experimental group." (p. 225) 2) "All of the pre-tests and the post-tests used in the study were domain-referenced tests that covered the content domains targeted prior to the study in the case of the pre-tests and during the period of investigation in the case of the post-tests." (p. 227) 3) "First, the teachers of each grade level assessed the relevance of the pre-test and post-test items to the curriculum objectives. Second, the two coordinators of English language were asked to do the same." (p. 227) Detailed Analysis: The outcome instruments were bespoke, researcher- and teacher-developed, domain-referenced tests built specifically to match the nine instructional units taught in this particular study. Their validity was checked internally (teacher and coordinator judgement), not by reference to any existing, independently normed, widely used standardized exam. There is no mention of a recognised national or commercial standardized assessment being used. Final sentence: Criterion E is not met because the pre- and post-tests were custom-built for this study rather than standardized, widely recognised exams.
  • T

    Term Duration

    • The intervention and outcome measurement together spanned only 6 weeks, well short of one academic term.
    • "The treatment lasted for 6 weeks and consisted of nine instructional units..." (p. 227)
    • Relevant Quotes: 1) "The treatment lasted for 6 weeks and consisted of nine instructional units, three units for each grade level." (p. 227) 2) "Finally, all students in the control and experimental classes took three developmentally appropriate post-tests that consisted of objective items that measured content covered during the 6-week period of experimentation." (p. 229) Detailed Analysis: The intervention began and ended within a single 6-week window, and the post-tests measuring outcomes were administered immediately at the close of this 6-week period. There is no indication of any delayed or follow-up measurement extending the tracking window closer to a full academic term (3-4 months). Six weeks is well short of one term. Final sentence: Criterion T is not met because outcomes were measured immediately after a 6-week intervention, far short of a full academic term.
  • D

    Documented Control Group

    • The control group's size, teachers, content, and baseline performance are clearly documented and shown to be comparable to the treatment group.
    • "Thus, the control group included 157 students and the experimental group included 161 students." (p. 225)
    • Relevant Quotes: 1) "Thus, the control group included 157 students and the experimental group included 161 students." (p. 225) 2) "In every grade level, the same teacher taught one control and one experimental class... these teachers were experienced with an average of 6.3 years of full-time service." (p. 227) 3) "Meanwhile, the control classes were taught the same content by the same teachers, but according to an individualistic approach whereby teachers presented the teaching points under study and required students to complete exercises in their regular textbooks." (p. 228) 4) "The results of the first t-test that compared the linguistic achievement of all participants in the control and experimental groups indicated that both groups did not differ significantly on the pre-test, t(316) = 0.57 P > 0.05. The means of the control and experimental groups were 59.5 (SD = 17.90) and 60.63 (SD = 17.15), respectively." (p. 230) Detailed Analysis: The paper documents the control group's size (157 students across the same three grade levels), the identity and experience of the teachers who taught it, exactly what instructional content and method it received (same nine units, individualistic textbook exercises), and its baseline (pre-test) equivalence with the experimental group. This is a clear, detailed description sufficient to assess comparability. Final sentence: Criterion D is met because the control group's size, teachers, instructional content and baseline performance are all explicitly documented and shown to be comparable to the experimental group.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was at the class level within the same two school branches, not at the level of separate schools.
    • "...enrolled in 12 ESL classes in two branches of a junior high school..." (p. 225)
    • Relevant Quotes: 1) "The participants were all the fourth-, fifth-, and sixth-grade students enrolled in 12 ESL classes in two branches of a junior high school (n = 318 students) in a Middle Eastern country." (p. 225) 2) "The 12 intact classes were randomly assigned to 6 control and 6 experimental conditions." (p. 225) Detailed Analysis: Both the control and experimental conditions were implemented as intact classes within the very same two school branches; the paper gives no indication that whole schools (or branches) were themselves the unit of assignment. Instead, individual classes within each of the two branches were split between conditions. This is class-level, not school-level, randomisation. Final sentence: Criterion S is not met because randomisation occurred among classes within the same two school branches, not among separate schools.
  • I

    Independent Conduct

    • The researchers who designed the STAD intervention also trained teachers and directly supervised the study's implementation, with no independent evaluator.
    • "...under the supervision of two coordinators and the researchers..." (p. 228)
    • Relevant Quotes: 1) "The teachers then observed the five components in action as demonstrated by one of the researchers, following which they developed and successfully demonstrated lesson plans applying STAD to their own curriculum." (p. 228) 2) "...they started teaching the instructional units to the experimental classes according to the dynamics of STAD and under the supervision of two coordinators and the researchers..." (p. 228) 3) "In addition, the researchers and coordinators visited every class during the period of experimentation in order to ensure proper implementation of the treatment." (p. 227) Detailed Analysis: The same researchers who designed and are known proponents of cooperative learning/STAD (one author has a prior publication on cooperative learning teacher training) personally trained the teachers, demonstrated the method, and directly supervised implementation and classroom visits throughout the study. There is no mention of any external, third- party evaluator independent of the intervention's designers who conducted data collection or analysis. Final sentence: Criterion I is not met because the same researchers who designed the intervention also trained teachers, supervised implementation, and presumably conducted the analysis, with no independent evaluator involved.
  • Y

    Year Duration

    • The study lasted only 6 weeks, far short of a full academic year, and criterion T was already not met.
    • "The treatment lasted for 6 weeks..." (p. 227)
    • Relevant Quotes: 1) "The treatment lasted for 6 weeks and consisted of nine instructional units, three units for each grade level." (p. 227) 2) "Finally, all students in the control and experimental classes took three developmentally appropriate post-tests that consisted of objective items that measured content covered during the 6-week period of experimentation." (p. 229) Detailed Analysis: Per the ERCT specification, if the weaker Term Duration criterion (T) is not met, the stronger Year Duration criterion (Y) cannot be met either. In addition, the study's actual duration of 6 weeks is nowhere near 75% of an academic year (~9-10 months). Final sentence: Criterion Y is not met, both because the study lasted only 6 weeks and because criterion T was already not met.
  • B

    Balanced Control Group

    • Both groups received the same content, teachers, and duration; only the instructional method (cooperative vs. individualistic) differed, so no resource imbalance is present.
    • "Meanwhile, the control classes were taught the same content by the same teachers..." (p. 228)
    • Relevant Quotes: 1) "The treatment lasted for 6 weeks and consisted of nine instructional units, three units for each grade level. These units were chosen from the regular English program of the subjects..." (p. 227) 2) "Meanwhile, the control classes were taught the same content by the same teachers, but according to an individualistic approach whereby teachers presented the teaching points under study and required students to complete exercises in their regular textbooks." (p. 228) 3) "In every grade level, the same teacher taught one control and one experimental class." (p. 227) Detailed Analysis: Applying the decision procedure: both conditions covered identical content (the same nine instructional units drawn from the regular English program), were taught by the very same teachers, and ran over the identical 6-week period. The only difference between conditions is the instructional method (small-group STAD activities vs. individual textbook exercises), not the quantity of class time, budget, or materials devoted to each group. The 4-day teacher training in STAD was given to the same six teachers who taught both the experimental and control sections, so it functioned as preparation for delivering the method rather than an extra resource channelled only to experimental students. There is no indication that experimental classes received additional instructional time, funding, or materials beyond what control classes received in their equivalent lesson periods, and the researchers' classroom monitoring visits covered both conditions equally. Final sentence: Criterion B is met because no extra instructional time, budget, or materials were provided to the experimental group beyond what the control group received; only the instructional method differed.

• Level 3 Criteria

  • R

    Reproduced

    • No evidence of an independent replication of this specific study was found in the paper or via external search.
    • Relevant Quotes: No quotes in the paper itself claim or reference any independent replication of this specific study. Detailed Analysis: An internet search for independent replications of this specific study (Ghaith & Yaghi, 1998, "Effect of cooperative learning on the acquisition of second language rules and mechanics," System 26) did not surface any peer-reviewed study by a different research team that replicated this particular STAD-vs-individualistic-instruction design for ESL grammar/mechanics acquisition in a different context. A check of the works citing this paper (via OpenAlex, 46 citing works) surfaced only broader cooperative learning literature and the original author's own later studies (e.g., Ghaith, 2002, "The relationship between cooperative learning, perception of social support, and academic achievement," System 30(3); Ghaith, 2003, "Effects of the Learning Together Model of Cooperative Learning on English as a Foreign Language Reading Achievement..." Bilingual Research Journal 27(3)), none of which constitute a direct independent reproduction of this study's specific design and outcome measures by a different research team. Final sentence: Criterion R is not met because no independent replication of this specific study was found in the paper or via external search.
  • A

    All-subject Exams

    • Only ESL rules and mechanics were assessed, and since criterion E is not met, criterion A cannot be met.
    • Relevant Quotes: 1) "Students were pre-tested and post-tested on their knowledge of ESL rules and mechanics." (p. 223, Abstract) 2) "These tests measured content learned during the period of investigation..." (p. 227) Detailed Analysis: The study assessed only ESL grammar rules and mechanics; no other core subjects (e.g., mathematics, science) were measured. In addition, per the ERCT specification, criterion A requires criterion E (standardised exam-based assessment) to be met as a prerequisite; since E was not met, A cannot be met regardless of subject coverage. Final sentence: Criterion A is not met because only a single subject (ESL rules/mechanics) was assessed and because criterion E was not met.
  • G

    Graduation Tracking

    • No follow-up beyond the immediate post-test was conducted or found via external search, and criterion Y was already not met.
    • Relevant Quotes: 1) "Finally, all students in the control and experimental classes took three developmentally appropriate post-tests that consisted of objective items that measured content covered during the 6-week period of experimentation." (p. 229) 2) "Further research that employs longer treatment periods is needed in order to determine if longer exposure to cooperative learning methods would increase the linguistic gains of second language learners..." (p. 232) Detailed Analysis: Measurement stopped immediately at the end of the 6-week intervention with no further follow-up, and the authors themselves call for future research with longer treatment periods, implying no such long-term or graduation tracking was conducted here. An internet search of works citing this paper and of the original authors' subsequent publications found no follow-up study tracking this specific cohort of 318 fourth-, fifth-, and sixth-graders toward graduation. Per the ERCT specification, since criterion Y was not met, criterion G is automatically not met as well. Final sentence: Criterion G is not met because tracking stopped immediately after the 6-week intervention, with no follow-up toward graduation found in the paper or via external search, and because criterion Y was not met.
  • P

    Pre-Registered

    • No mention of a pre-registered protocol or registry is found anywhere in the paper or via external search.
    • Relevant Quotes: No quotes in the paper mention any study registry, pre-registration platform, or pre-specified analysis plan published before data collection began. Detailed Analysis: The paper (published 1998) contains no statement about pre-registration of hypotheses, methods, or analysis plans prior to data collection. Formal pre-registration registries for educational/social science research (e.g., OSF, AEA RCT Registry) did not exist or were not in standard use at the time of this study, and no such reference appears anywhere in the text or was found via external search. Final sentence: Criterion P is not met because no pre-registration of the study protocol is mentioned anywhere in the paper or found via external search.