Processing Instruction Versus Traditional Instruction: The Transfer-of-Training Effects on Chinese EFL Learners

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the class level (three intact classes assigned to three conditions), which satisfies the class-level RCT requirement.
    • "Two of the classes were randomly assigned to one of two experimental conditions: Processing Instruction (PI) and Traditional Instruction (TI). And the remaining one is the control group." (p. 5)
    • Relevant Quotes: 1) "This study featured the pretest-immediate-posttest design, with 106 EFL learners from three classes. Two of the classes were randomly assigned to one of two experimental conditions: Processing Instruction (PI) and Traditional Instruction (TI). And the remaining one is the control group." (p. 5) 2) "In the second research session, a random draw was conducted to determine which group corresponded to which therapy." (p. 7) 3) "Three instructional methods were used in three groups randomly: the PI group (N = 36), the TI group (N = 35), and the control group (N = 35)." (p. 5) Detailed Analysis: The ERCT 'C' criterion requires randomisation at the class level (or stronger), rather than assignment of individual students within a single class. The paper explicitly describes a random draw that determined which of the three intact classes received which of the three conditions (PI, TI, or control). This means the unit of randomisation was the whole class, not individual students within one class, so contamination between conditions within a classroom is avoided. The fact that there was only one class per condition limits statistical power and replication within condition, but it does not change the unit of randomisation, which is what this criterion evaluates. Criterion C is met because randomisation was conducted at the level of entire, intact classes rather than at the individual-student level within a single class.
  • E

    Exam-based Assessment

    • The tests were custom-built instruments derived from the treatment activities, not a widely recognised standardised exam.
    • "The multiple-choice questions were derived from the referential activity used in PI group." (p. 7)
    • Relevant Quotes: 1) "The multiple-choice questions were derived from the referential activity used in PI group." (p. 7) 2) "The cloze questions, which was derived from the an activity in TI group, included only five target questions." (p. 7) 3) "All items in tests were adopted form those two authoritative grammar dictionaries and exam questions of previous final exams or senior high school entrance examinations for the whole city. The reliability and validity of those tests had been verified in the pilot study, as well as the difficulty of those tests." (p. 7) Detailed Analysis: Criterion E requires a standardised, widely recognised exam rather than a bespoke instrument built for the study. Here, the interpretation-task items (multiple choice, true/false) were explicitly derived from the referential activities used inside the PI treatment itself, and the production-task cloze items were derived from an activity used in the TI treatment. Although individual sentence items were sourced from grammar dictionaries and past local exam questions, the overall instrument (pretest/posttest/delayed-posttest battery targeting three specific grammar structures) was assembled and piloted by the research team specifically for this study; it is not an existing, externally validated standardised exam administered as-is. Note that a near-identical sentence about the dictionaries and exam questions also appears on p. 6 (in the Target Linguistic Items section, worded "referenced to" rather than "adopted form"), but the quote above verbatim matches the p. 7 Assessment Tasks passage. Criterion E is not met because the assessment was a custom-built instrument derived from the treatment activities themselves, not a standardised exam.
  • T

    Term Duration

    • The intervention and follow-up together spanned only about two to three weeks, far short of the one-term minimum required by this criterion.
    • "Two weeks later, delayed post-tests were administered, which focused on the same three linguistic items." (p. 7)
    • Relevant Quotes: 1) "All participants were required to participate in all tests (pretest, immediate posttest, delayed posttest) and treatment for 2 hr, which were divided into three lessons." (p. 5) 2) "After completing the entire teaching method, students underwent three immediate post-tests on the primary target language item and the two transfer-of-training linguistic items. Two weeks later, delayed post-tests were administered, which focused on the same three linguistic items." (p. 7) 3) "Basically, the study only had 106 students... Also, the teaching time for both PI and TI was only 2 hours, which might not have been long enough for students to really learn the grammar well." (Plain Language Summary, p. 1) Detailed Analysis: Criterion T requires outcomes to be measured at least one academic term (roughly 3-4 months) after the intervention begins. Here, the entire treatment consisted of only three lessons totalling 2 hours, and the final (delayed) outcome measurement occurred just 2 weeks after the immediate posttest, i.e. only about 2-3 weeks after the intervention began. This falls far short of a full academic term, and the authors themselves flag the short instruction/tracking time as a limitation. Criterion T is not met because the interval from intervention start to final measurement was only about two to three weeks, far shorter than one academic term.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline performance, and the exact nature of what it received are all clearly documented in the paper.
    • "The control group in this study did not receive any explicit instruction on the target linguistic item. However, they were exposed to various texts, some of which contained sentences incorporating the target linguistic item." (p. 7)
    • Relevant Quotes: 1) "Table 2. Descriptive Statistics About Participants ... Con 35 21 14 12.26 ... total 106 65 41 12.25" (Table 2, p. 5) 2) "The control group in this study did not receive any explicit instruction on the target linguistic item. However, they were exposed to various texts, some of which contained sentences incorporating the target linguistic item." (p. 7) 3) "Those data revealed scores of the three groups were neck and neck in terms of not only interpretation tasks (F2,103 = 0.10, p = .90 > .05) but also the production task (F2,103 = 0.13, p = .88 > .05), which meant that the extents of acquisition for the third-person singular present tense in the pretest phase were nearly the same in these three groups." (p. 8) Detailed Analysis: Criterion D requires clear documentation of the control group's size, demographics, baseline performance, and the conditions it experienced. The paper reports the control group's exact size (N = 35), its gender split (21 male, 14 female) and average age (12.26 years) in Table 2, clearly describes what the control group did and did not receive (no explicit grammar instruction, but reading/listening exposure), and statistically confirms via one-way ANOVA that the control group did not differ from the treatment groups at baseline. Criterion D is met because the control group's size, demographics, baseline scores, and treatment conditions are all clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Only one school was involved, with randomisation at the class level within it, so the school-level RCT requirement is not satisfied.
    • "The original participants were from Grade 7 at a junior high school affiliated with Hunan University in Changsha." (p. 5)
    • Relevant Quotes: 1) "The original participants were from Grade 7 at a junior high school affiliated with Hunan University in Changsha." (p. 5) 2) "This study featured the pretest-immediate-posttest design, with 106 EFL learners from three classes." (p. 5) Detailed Analysis: Criterion S requires randomisation among whole schools rather than among classes or students within a single school. This study drew its 106 participants from three classes within a single junior high school affiliated with Hunan University; the random draw allocated these three intact classes (not separate schools) to the three conditions. No other schools were involved, and there is no description of school-level allocation. Criterion S is not met because randomisation occurred among classes within a single school, not among schools.
  • I

    Independent Conduct

    • No statement in the paper indicates that data collection, instruction, or analysis were conducted by an independent party separate from the study designers.
    • Relevant Quotes: 1) "This study was approved by Foreign Studies College Research Ethical Review Board of Hunan University." (Ethical Approval and Informed Consent, p. 13) 2) "TI Treatment. There were three periods in TI treatment. As shown in Figure 2, teachers illustrated the definition and structures of the target linguistic item..." (p. 6) 3) "PI Treatment. There were also three periods (40 min for each lesson) in PI treatment to teach English third person singular present tense." (p. 6) Detailed Analysis: Criterion I requires that the study be conducted independently of those who designed the intervention, typically via an external evaluation team. Throughout the Methods, Treatment, and Data Analysis sections, the paper gives no indication that any external or third-party team designed materials, delivered instruction, administered tests, or analysed the data. Instead, the description reads as though the same research team (affiliated with Hunan University, which also granted ethical approval) designed the PI and TI materials, delivered the lessons, and performed the statistical analysis in SPSS. No disclosure statement, acknowledgement, or methods text mentions an independent or blinded evaluator. Criterion I is not met because there is no evidence of independent conduct separate from the team that designed the intervention.
  • Y

    Year Duration

    • The study tracked outcomes for only about two to three weeks in total, far short of the year-long requirement, and criterion T (a prerequisite) was not met.
    • "Two weeks later, delayed post-tests were administered, which focused on the same three linguistic items." (p. 7)
    • Relevant Quotes: 1) "All participants were required to participate in all tests (pretest, immediate posttest, delayed posttest) and treatment for 2 hr, which were divided into three lessons." (p. 5) 2) "Two weeks later, delayed post-tests were administered, which focused on the same three linguistic items." (p. 7) Detailed Analysis: Criterion Y requires outcomes to be tracked for at least 75% of an academic year, and by the standard's own rule, if criterion T (term duration) is not met, Y cannot be met either. Here the total span from the start of the intervention to the final measurement was only about two to three weeks, nowhere near the required academic-year timescale, and criterion T was already found not met above. Criterion Y is not met, both because criterion T is not met and because the actual measured interval (about two to three weeks) is far shorter than 75% of an academic year.
  • B

    Balanced Control Group

    • The withheld instructional resource is the explicit treatment variable under investigation (instruction vs. no instruction), and item exposure was otherwise kept equivalent across groups, so the control's business-as-usual status does not violate this criterion.
    • "The control group, on the other hand, would not receive any formal instruction but will only be given some reading tasks." (p. 6)
    • Relevant Quotes: 1) "To ensure that all three experimental groups would receive equal and appropriately challenging instructional activities, each lesson would contain six tokens with similar themes. The control group, on the other hand, would not receive any formal instruction but will only be given some reading tasks." (p. 6) 2) "The control group in this study did not receive any explicit instruction on the target linguistic item. However, they were exposed to various texts... After listening and reading, learners were required to complete writing tasks and answer comprehension questions related to these passages." (p. 7) 3) "The total number of linguistic items presented in the control group was equivalent to that in the PI and TI groups." (p. 7) 4) "Although different teaching methods have a collorary of different teaching materials more or less, the difficulty of the teaching materials, the duration of the treatment time, the number of instructional activities, and the time allotted for examinations were all kept as consistent as possible." (p. 7) Detailed Analysis: Between the two active treatment arms (PI and TI), the paper explicitly states that treatment duration, difficulty, and number of activities were kept as consistent as possible (quote 4 above; this sentence follows the description of PI/TI post-testing and is read as referring to the two instructed arms, since the control group has no "teaching materials" of its own), so there is no imbalance between PI and TI. The more delicate question is the control group, which received no explicit grammar instruction. However, the study's design explicitly frames "explicit instruction" itself as the variable under investigation: the control condition is deliberately the "business-as-usual"/ no-instruction baseline used to show that PI and TI produce effects beyond mere exposure (Tables 4-6 compare both instructed groups against this control). The authors also state they matched the total amount of linguistic-item exposure (reading/listening texts containing the target forms) across all three groups, withholding only the explicit teaching component that is the explicit treatment variable being tested. Under the ERCT decision rule, when the extra resource (here, explicit instruction) is itself the treatment variable being investigated, the control may legitimately lack that resource as long as this is the clearly stated intent, which is the case here. Criterion B is met because the withheld resource (explicit grammar instruction) is the explicit treatment variable under study, the control's business-as-usual status is clearly intentional, and exposure to the linguistic items was otherwise kept equivalent across groups.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study by a different research team was found in the paper or via internet search.
    • Relevant Quotes: 1) No quote found in the paper's introduction, discussion, or references referring to any subsequent independent replication of this specific experiment (comparing PI, TI, and control on third-person singular present tense, simple past tense, and passive voice in Chinese Grade 7 EFL learners). Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different research team, published in a peer-reviewed journal. This is a 2024 paper, and nothing in the text refers to any subsequent independent replication of this particular experiment. The paper does situate itself among many related PI studies (e.g., Benati & Lee, 2008; Chan, 2018; Ma, 2017) that examine similar questions in other populations, but these are prior, related studies rather than replications of this specific study. An internet search (web search for the article title, its DOI 10.1177/21582440241231037, and combinations of the authors' names with "replication") was conducted to check for post-publication replications. No paper by a different research team replicating this specific Chinese Grade 7 PI/TI/control design was found. A later 2024 SAGE Open article by the same lead author, Tao Zeng (with Peng Xu and Xi Gao), "The Effect of Explicit Information in Processing Instruction on Middle School Students' Acquisition of English Passive Voice," was located, but it is a different study by an overlapping (not independent) author team and does not replicate this specific experiment, so it does not satisfy this criterion. Criterion R is not met because no independent replication of this specific study was found via internet search.
  • A

    All-subject Exams

    • Only three specific English grammar structures were assessed, not all main subjects, and criterion E (a prerequisite) was not met.
    • "The three target linguistic items were English third-person singular present tense, simple past tense, and passive voice." (p. 5)
    • Relevant Quotes: 1) "The three target linguistic items were English third-person singular present tense, simple past tense, and passive voice." (p. 5) 2) "Every participant was required to finish three tests, each test containing all the three linguistic items, pretest, immediate posttest, and delayed posttest, for third person singular simple present, simple past, and be-passives..." (p. 7) Detailed Analysis: Criterion A requires assessment of all main subjects, and by the standard's own rule, if criterion E is not met, criterion A cannot be met either. Here, the assessment was restricted to three specific English grammar structures within a single subject (English/L2 grammar); no other core school subjects (e.g., mathematics, science, first-language literacy) were assessed at all, and criterion E was already found not met above. Criterion A is not met, both because criterion E is not met and because only a narrow set of English grammar structures, not all main subjects, were assessed.
  • G

    Graduation Tracking

    • Follow-up stopped two weeks after the intervention with no tracking toward graduation, no subsequent follow-up publication was found, and criterion Y (a prerequisite) was not met.
    • "This study almost only focuses on the period from immediate posttest to delayed posttest and does not pay attention to the long-term primary effects and TOT effects of PI and TI." (p. 12)
    • Relevant Quotes: 1) "Two weeks later, delayed post-tests were administered, which focused on the same three linguistic items." (p. 7) 2) "Finally, due to objective factors, this study almost only focuses on the period from immediate posttest to delayed posttest and does not pay attention to the long-term primary effects and TOT effects of PI and TI, which are also seldom studied by scholars." (Limitations, p. 12) Detailed Analysis: Criterion G requires tracking of participants until graduation, and by the standard's own rule, if criterion Y is not met, criterion G cannot be met either. Here, tracking stopped at a delayed posttest just 2 weeks after the immediate posttest; the authors explicitly note in their Limitations section that the study does not address long-term effects, and criterion Y was already found not met above. An internet search for follow-up publications by the same author team (Tao Zeng, Chang Xu, Jia Hu, Xiuzhi Fu) tracking this same cohort of Grade 7 students toward graduation was conducted. No such follow-up study was found; the only other located paper by the lead author, Tao Zeng, is a distinct 2024 study on a different sample (passive voice, with different co-authors Peng Xu and Xi Gao) rather than a graduation-tracking follow-up of this cohort. Criterion G is not met because data collection stopped after a two-week delayed posttest with no follow-up publication tracking this cohort located, and criterion Y (a prerequisite) was not met.
  • P

    Pre-Registered

    • The paper contains no statement, link, or date indicating that the study protocol was pre-registered before data collection began, and no registry record was found via internet search.
    • Relevant Quotes: 1) "Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest..." (p. 13) 2) "Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article." (p. 13) 3) "Ethical Approval and Informed Consent: This study was approved by Foreign Studies College Research Ethical Review Board of Hunan University." (p. 13) Detailed Analysis: Criterion P requires a publicly pre-registered protocol (hypotheses, methods, planned analyses) dated before data collection began. The paper includes standard disclosure sections (conflicts of interest, funding, ethical approval) but contains no statement anywhere about registering the study protocol on a registry platform, no registration ID, and no registration date. Ethical approval is not the same as pre-registration of the research plan. An internet search of common pre-registration registries (e.g., OSF Registries, AsPredicted, ClinicalTrials.gov) combined with the authors' names and the study topic was conducted, and no pre-registration record for this study was found. Criterion P is not met because no pre-registration statement, registry link, or registration date is provided anywhere in the paper, and none was found via internet search.