Timing of form-focused instruction: Effects on EFL learners' grammar learning

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Whole intact classes were randomly selected and each entire class received one condition, so allocation was at the class level rather than within classes.
    • "They came from four classes which were randomly chosen among 12 eighth-grade classes."
    • Relevant Quotes: 1) "A total of 169 Chinese middle school learners were assigned to four conditions randomly: control, before-isolated FFI, integrated FFI, and after-isolated FFI." (Abstract, p. 405) 2) "The participants were 169 students attending a Chinese middle school. They came from four classes which were randomly chosen among 12 eighth-grade classes." (p. 410) 3) "Three classes were treated as experimental groups and received integrated, before-isolated or after-isolated FFI, respectively. The fourth class was treated as a control group." (p. 410) 4) "The four classes were at a similar level, with no significant differences among the scores of their midterm examination: F(3, 165) = 1.975, p = .120" (p. 410) Detailed Analysis: The unit of treatment allocation was the intact class, not individual students within one classroom. Four whole classes were randomly chosen from the twelve eighth-grade classes in the school, and each entire class received one of the four conditions; the abstract explicitly states that assignment to the four conditions was random. Because each condition was delivered to a separate intact class, contamination between treatment and control students within a classroom is avoided, which is the core concern of criterion C. The design is weak in that only four classes were involved (one class per condition) and the exact mechanics of assigning conditions to classes are only briefly described, but the paper does describe class-level random selection and states random assignment to conditions, satisfying the class-level requirement. Criterion C is met because randomisation and treatment allocation operated on whole classes rather than on individual students within a single class.
  • E

    Exam-based Assessment

    • Outcomes were measured with custom researcher-made error correction and picture writing tests, not widely recognised standardised exams.
    • "The findings are triangulated through two tests (Cohen et al., 2017): picture writing test (PWT) and error correction test (ECT) (see Appendix C)."
    • Relevant Quotes: 1) "A picture writing test and a written error correction test were employed to measure students' performance." (Abstract, p. 405) 2) "The findings are triangulated through two tests (Cohen et al., 2017): picture writing test (PWT) and error correction test (ECT) (see Appendix C)." (p. 414) 3) "The ECT included different versions for pretest, immediate and delayed posttests. The test contained 30 sentences, of which 24 sentences examined the passive voice and six sentences were distractors" (p. 414) 4) "Following Lee (2007) and Li et al. (2016b) as well as the pilot study with ten students of similar proficiency level from another eighth-grade class, the target items were also equally distributed among three types of errors" (p. 414) 5) "The PWT also had three versions that were used for pretest and posttests. The test contained eight pictures with eight keywords (e.g., send, return) presented on each picture" (p. 414) Detailed Analysis: Criterion E requires the use of widely recognised standardised exams rather than instruments created specifically for the study. Both outcome measures here (the error correction test and the picture writing test) were custom-made by the researchers for this experiment, constructed following prior research designs and validated only through a small pilot with ten students and internal reliability statistics (Cronbach's alpha). Neither test is a national, state-wide, or otherwise widely recognised standardised examination. The only standardised-style measure mentioned, the midterm examination, was used solely to check baseline equivalence, not as an outcome. Criterion E is not met because outcomes were measured with researcher-designed tests, not recognised standardised exams.
  • T

    Term Duration

    • Outcomes were measured three hours and two weeks after a 90-minute intervention, far short of the required one-term follow-up.
    • "Considering that recency effect may occur in the after-isolated group, the posttests were administered three hours after instructional treatment. Two weeks later, all groups finished the delayed posttests."
    • Relevant Quotes: 1) "All the four groups first took the pretests. One week later, instructional treatments were carried out for the three experimental groups." (p. 410) 2) "instructional treatment for each experimental group was designed to last two consecutive class hours, with a 10-minute break in between." (p. 410) 3) "Considering that recency effect may occur in the after-isolated group, the posttests were administered three hours after instructional treatment. Two weeks later, all groups finished the delayed posttests." (p. 410) 4) Figure 1 shows: "Pretest Week 1 ... Treatment / Immediate posttest Week 2 ... Delayed posttest Week 4" (p. 411) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (roughly 3-4 months) after the intervention begins. Here the intervention itself consisted of a single session of two consecutive 45-minute class hours in Week 2, the immediate posttest was administered three hours after the treatment, and the final delayed posttest followed only two weeks later in Week 4. The entire span from intervention start to the last outcome measurement is approximately two weeks, far short of an academic term. The authors themselves acknowledge that "the instructional treatment for each experimental group is relatively short" (p. 423). Criterion T is not met because the interval from intervention start to the final measurement was about two weeks, well below one academic term.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline equivalence and no-treatment condition are documented in the text and tables.
    • "The four classes were at a similar level, with no significant differences among the scores of their midterm examination: F(3, 165) = 1.975, p = .120; error correction pretest, F(3, 165) = 1.883, p = .134 and picture writing pretest, F(3, 165) = 1.594, p = .193."
    • Relevant Quotes: 1) "The fourth class was treated as a control group." (p. 410) 2) "The four classes were at a similar level, with no significant differences among the scores of their midterm examination: F(3, 165) = 1.975, p = .120; error correction pretest, F(3, 165) = 1.883, p = .134 and picture writing pretest, F(3, 165) = 1.594, p = .193." (p. 410) 3) "The learners were aged between 13 and 14, and they had learned English for nearly six years. They attended five English lessons on a weekly basis with each lesson lasting 45 minutes. All participants were native Chinese speakers with limited exposure to English." (p. 410) 4) "The control group finished all tests without any treatment." (p. 410) 5) "Control group (N = 42) Pretest 2.05 ... Posttest 3.33 ... Delayed posttest 4.29" (Table 1, p. 415) Detailed Analysis: Criterion D requires clear documentation of the control group's composition, baseline performance, and conditions. The paper specifies the control group's size (N = 42), its demographic profile (age 13-14, native Chinese speakers, about six years of English study, five 45-minute English lessons weekly), its baseline equivalence with the treatment groups on the midterm examination and both pretests, and the condition it experienced (it completed all tests but received no instructional treatment). Tables 1, 2, 4 and 5 report full descriptive statistics for the control group at pretest, posttest and delayed posttest. This is sufficient documentation for proper comparison. Criterion D is met because the control group's size, characteristics, baseline scores and treatment conditions are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The trial was conducted among four classes within a single middle school, so there was no school-level randomisation.
    • "The participants were 169 students attending a Chinese middle school. They came from four classes which were randomly chosen among 12 eighth-grade classes."
    • Relevant Quotes: 1) "The participants were 169 students attending a Chinese middle school. They came from four classes which were randomly chosen among 12 eighth-grade classes." (p. 410) 2) "Three classes were treated as experimental groups and received integrated, before-isolated or after-isolated FFI, respectively. The fourth class was treated as a control group." (p. 410) Detailed Analysis: Criterion S requires randomisation among schools or equivalent implementing institutions. This study took place within a single Chinese middle school, and the units of allocation were four classes inside that one school. No multiple schools, sites or centres were involved, and no school-level randomisation is described anywhere in the paper. Criterion S is not met because randomisation occurred among classes within a single school, not among schools.
  • I

    Independent Conduct

    • The authors designed the intervention, the tests and the analysis themselves with no external or independent evaluation team.
    • "all the instructional treatments were planned and scripted and the teacher was asked to strictly follow the predetermined teaching procedure."
    • Relevant Quotes: 1) "To avoid different results caused by different teachers' teaching styles, all experimental treatments were conducted by one teacher who had 14 years' teaching experience. She had not taught these students before." (p. 410) 2) "all the instructional treatments were planned and scripted and the teacher was asked to strictly follow the predetermined teaching procedure." (pp. 410-411) 3) "The video of the instruction was viewed by the researchers and the teacher, and it showed that the instruction matched the designed instructional treatments." (p. 411) 4) "Materials and activities used in the treatments were validated by two researchers and three EFL teachers" (p. 413) 5) "This work was supported by MOE (Ministry of Education in China) Project of Humanities and Social Sciences under Grant Number 21YJC740025." (Acknowledgment, p. 424) Detailed Analysis: Criterion I requires that the study be conducted independently of the intervention designers, for example by an external evaluation team. In this study the two authors designed the instructional treatments, scripted the lessons, developed the outcome tests, supervised implementation through video review, and performed all data analysis themselves. Although a classroom teacher who had not previously taught the students delivered the lessons, she followed scripts written by the researchers and worked under their direction; she was an implementer, not an independent evaluator. There is no statement of any third-party oversight, external data collection or independent analysis. Criterion I is not met because the same research team designed the intervention, developed the measures, and conducted and analysed the study without independent oversight.
  • Y

    Year Duration

    • The full study lasted about four weeks, far below the required 75% of an academic year of outcome tracking.
    • "Two weeks later, all groups finished the delayed posttests."
    • Relevant Quotes: 1) "instructional treatment for each experimental group was designed to last two consecutive class hours, with a 10-minute break in between." (p. 410) 2) "Two weeks later, all groups finished the delayed posttests." (p. 410) 3) "First and foremost, the instructional treatment for each experimental group is relatively short, which may be biased in favor of isolated FFI." (p. 423) Detailed Analysis: Criterion Y requires tracking of outcomes for at least 75% of an academic year from intervention start. Since criterion T (one term) is already not met, criterion Y cannot be met per the standard's dependency rule. The entire study spanned roughly four weeks from pretest to delayed posttest, with only about two weeks between the 90-minute intervention and the final measurement, which is a tiny fraction of an academic year. Criterion Y is not met because the study tracked outcomes for only about two weeks after a one-day intervention, nowhere near 75% of an academic year.
  • B

    Balanced Control Group

    • Treatments took place within regularly scheduled lessons with matched duration and intensity across groups, so no extra time or budget favoured the intervention groups.
    • "All the students and the teacher agreed to participate in the study, and the instructional treatments were conducted during their regularly scheduled English lessons."
    • Relevant Quotes: 1) "All the students and the teacher agreed to participate in the study, and the instructional treatments were conducted during their regularly scheduled English lessons." (p. 410) 2) "To ensure that the duration and intensity of FFI were the same in each group, all the instructional treatments were planned and scripted and the teacher was asked to strictly follow the predetermined teaching procedure." (pp. 410-411) 3) "the duration and intensity of FFI and communicative activities were consistent throughout all instructional treatments" (p. 413) 4) "The control group finished all tests without any treatment." (p. 410) 5) "They attended five English lessons on a weekly basis with each lesson lasting 45 minutes." (p. 410) Detailed Analysis: Applying the criterion B decision tree: did the intervention add extra time or budget beyond normal schooling? The instructional treatments were delivered during the students' regularly scheduled English lessons, so the experimental groups received no additional instructional time, materials requiring extra budget, or out-of-class resources; the treatment replaced ordinary lesson content within the normal timetable. All students in all four classes continued their standard five weekly 45-minute English lessons, and the control group simply completed the tests within that same normal schedule, receiving no compensating activity but also losing no time or resources relative to the intervention groups. Furthermore, the three experimental conditions, which constitute the study's central contrast (timing of FFI), were explicitly equalised in duration and intensity through scripted, video-checked lessons validated by five analysers. Under branch 1 of the decision tree (no extra time/budget present), the balance requirement is trivially satisfied because no group received additional time, staff attention, or materials budget relative to another. Criterion B is met because the intervention added no extra time or resources beyond regularly scheduled lessons, and the experimental conditions were strictly matched in duration and intensity.

• Level 3 Criteria

  • R

    Reproduced

    • The paper itself replicates earlier work by others, and internet search found no independent team having replicated this specific study.
    • Relevant Quotes: 1) "This study partially replicates the study of Spada et al. (2014) to examine how different FFI timing impacts Chinese middle school students, focusing on the passive voice." (p. 408) 2) "The present study attempts to verify and provide further empirical evidence about the effects of different timing of FFI on difficult grammatical feature" (p. 409) 3) "The finding that before-isolated FFI was the most beneficial for learners corroborates the results of a previous study (Xu & Li, 2021)." (pp. 421-422) Internet Research: A search for citations of this paper (Xu, J., & Li, C. (2022). Timing of form-focused instruction: Effects on EFL learners' grammar learning. Studies in Second Language Learning and Teaching, 12(3), 405-433, https://doi.org/10.14746/ssllt.2022.12.3.4) and for other studies replicating its specific three-way (integrated before-isolated, after-isolated) FFI-timing design with Chinese eighth-grade learners and the English passive voice did not surface any independent replication by a different research team. The closest related studies located are by the same authors: Xu, J. F., & Li, C. Y. (2021), "The effects of the timing of form-focused instruction on EFL learners' learning of difficult and easy grammatical features: A comparative study," System, 101, 102612 (an earlier, same-team study using different grammatical structures), and Li, C., & Xu, J. (2023), "The Sustainability of Form-Focused Instruction in Classrooms: Chinese Secondary School EFL Teachers' Beliefs and Practices," Sustainability, 15(7), 6109, which is a teacher-belief survey, not a replication of this experiment. No independent, peer-reviewed replication of this specific study by a different team was found. Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different research team in a different context and published in a peer-reviewed journal. This paper is itself a partial replication of Spada et al. (2014), which is the reverse of what the criterion asks for; being a replication of earlier work does not mean this study has been reproduced. The related prior study it corroborates (Xu & Li, 2021) is by the same authors and therefore not independent, and subsequent internet searching found no independent team having replicated this particular three-way FFI-timing experiment with Chinese adolescent learners. Criterion R is not met because no independent replication of this specific study by a different research team is reported in the paper or found via internet search.
  • A

    All-subject Exams

    • Only English passive-voice knowledge was tested with custom instruments, so neither standardised exams nor all-subject coverage is present.
    • "Second, only explicit knowledge and one grammatical feature were investigated."
    • Relevant Quotes: 1) "A picture writing test and a written error correction test were employed to measure students' performance." (Abstract, p. 405) 2) "Therefore, the present study focused on measuring students' explicit knowledge. The findings are triangulated through two tests (Cohen et al., 2017): picture writing test (PWT) and error correction test (ECT)" (pp. 413-414) 3) "Second, only explicit knowledge and one grammatical feature were investigated." (p. 423) Detailed Analysis: Criterion A requires standardised exam-based assessment across all main subjects taught at that school level. Criterion E is a prerequisite and is not met, since both outcome measures were custom-built research instruments. Moreover, the study assessed only one narrow feature of one subject: knowledge of the English passive voice. No other school subjects (Chinese, mathematics, science, etc.) were assessed, and the paper offers no specialised-education rationale of the kind the exception clause contemplates for upper secondary vocational contexts; this is a general middle school. Criterion A is not met because criterion E fails and only one grammatical feature of English was assessed rather than all main subjects.
  • G

    Graduation Tracking

    • Tracking ended two weeks after the intervention, with no follow-up of students until graduation found in the paper or via internet search.
    • "Two weeks later, all groups finished the delayed posttests."
    • Relevant Quotes: 1) "Two weeks later, all groups finished the delayed posttests." (p. 410) 2) "Different instructional treatment hours might produce different results, and this should be investigated in future research." (p. 423) Internet Research: A search was conducted for subsequent papers by Jinfen Xu and/or Changying Li that might track this same cohort of 169 eighth-grade students through to graduation. No such follow-up study was found. The only later related publication located is Li, C., & Xu, J. (2023), "The Sustainability of Form-Focused Instruction in Classrooms: Chinese Secondary School EFL Teachers' Beliefs and Practices," Sustainability, 15(7), 6109, which is a teacher-belief survey about classroom practice, not a cohort-tracking study of the students from this experiment, and it contains no data on this cohort's outcomes at graduation. Detailed Analysis: Criterion G requires tracking participants until graduation from their educational stage. Since criterion Y is not met, criterion G cannot be met under the standard's dependency rule. The eighth-grade participants were followed for only two weeks after the intervention; there is no mention in the paper of any follow-up to the end of middle school (ninth grade graduation) or beyond, and internet search found no published follow-up study tracking this specific cohort to graduation. Criterion G is not met because measurement stopped two weeks after the intervention, criterion Y is not met, and no graduation-tracking follow-up publication for this cohort was found.
  • P

    Pre-Registered

    • The paper contains no reference to any pre-registered protocol or registry entry, and none was found online.
    • Relevant Quotes: 1) "This work was supported by MOE (Ministry of Education in China) Project of Humanities and Social Sciences under Grant Number 21YJC740025." (Acknowledgment, p. 424) 2) No statement referencing any trial registry, pre-registration platform, registration ID, or pre-registered protocol appears anywhere in the paper. Internet Research: A search for a pre-registration of this study (using the authors' names, the grant number 21YJC740025, and the study topic) on registries such as OSF, AsPredicted, and ClinicalTrials.gov did not locate any pre-registration record. The grant number identifies a funding project, not a study pre-registration. Detailed Analysis: Criterion P requires that the full study protocol, including hypotheses, methods and planned analyses, be registered on a public registry before data collection began. The paper contains no mention of pre-registration, no registry name or identifier (e.g., ClinicalTrials.gov, OSF, AsPredicted), and no registration date. The only administrative detail provided is the funding grant number, which is not a pre-registration, and internet search found no corresponding registry entry. Criterion P is not met because no pre-registration of the study protocol is mentioned in the paper or found via internet search.