Processing instruction in helping map forms and meaning in second language acquisition of English simple past

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Individual students, not intact classes or schools, were randomly assigned to the three instructional groups, and this was not a one-to-one tutoring intervention, so the class-level exception does not apply.
    • "There were 66 students assigned to one of the three groups randomly: PI group (n = 21), TI group (n = 24), and II group (n = 21)." (p. 722)
    • Relevant Quotes: 1) "There were 66 students assigned to one of the three groups randomly: PI group (n = 21), TI group (n = 24), and II group (n = 21)." (p. 722) 2) "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days." (p. 722) 3) "The experimental groups were taught by three different teachers to avoid the Hawthorne effect, or so-called observer effect, which is the modification of behavior by the students who might be aware of being observed." (p. 722) Detailed Analysis: The ERCT 'C' criterion requires randomisation at the class (or stronger school) level, unless the intervention is a one-to-one tutoring/personal teaching design, in which case student-level assignment is acceptable. Here, 66 individual primary 2 students who scored 60% or below on a pretest were drawn together and then randomly assigned as individuals to one of three newly formed groups (PI, TI, II), each subsequently taught as a group by a separate teacher. This is not personal one-to-one tutoring; it is small-group classroom-style teaching (21-24 students per group), so the tutoring exception does not apply. Because the unit of randomisation was the individual student rather than a pre-existing intact class or school, the design mirrors the "Example 1" case in the ERCT guidance (Casselman et al.), where individual-student randomisation into several study groups was judged not to meet this criterion. Final sentence: Criterion C is not met because randomisation was carried out at the individual student level rather than the class or school level, and no tutoring exception applies.
  • E

    Exam-based Assessment

    • Outcomes were measured with researcher-designed interpretation and production tasks created specifically for this study, not a widely recognised standardised exam.
    • "The pretest and posttest consisted of five tasks (both interpretation and production tasks)—a sentence-level interpretation task, two discourse-level interpretation tasks, a fill-in-the-blanks task, and a sentence creation task—to determine the participants' ability of producing sentences using English simple past." (p. 723)
    • Relevant Quotes: 1) "The pretest and posttest consisted of five tasks (both interpretation and production tasks)—a sentence-level interpretation task, two discourse-level interpretation tasks, a fill-in-the-blanks task, and a sentence creation task—to determine the participants' ability of producing sentences using English simple past." (p. 723) 2) "Assessment tasks in the study included an interpretation task of simple sentences (some in English simple present and some in English simple past) without temporal adverbs, requiring the participants to listen to sentences and then indicate if what they heard was in simple present or simple past." (p. 721, describing the paradigm the author's own tasks were modelled on) Detailed Analysis: The outcome measures (sentence-level and discourse-level interpretation tasks, fill-in-the- blanks, sentence creation) were purpose-built by the researcher to probe the specific linguistic target of the study (English simple past, interpretation versus production). There is no mention anywhere in the Method section of a nationally or internationally standardised, widely recognised exam (e.g., a school curriculum exam or an established language proficiency test) being used for the outcome measure. This is precisely the kind of custom-built, intervention-aligned assessment the ERCT standard's E criterion is designed to flag, since it may be overly tailored to the intervention and inflate apparent effects. Final sentence: Criterion E is not met because the study relied on custom-made interpretation and production tasks rather than a standardised, widely-recognised exam.
  • T

    Term Duration

    • The intervention and its only posttest both occurred within a single week, far short of a full academic term, with no delayed posttest possible.
    • "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days. Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722)
    • Relevant Quotes: 1) "A pretest was given to the learners one week before the lessons and a posttest immediately after the lessons." (p. 723) 2) "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days. Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722) Detailed Analysis: The intervention itself consisted of three consecutive days of 1.5-hour lessons (4.5 hours total), and outcomes were measured with an immediate posttest right after the last lesson, following a pretest given only one week earlier. The authors explicitly state that no delayed posttest was possible due to curriculum constraints. This entire window (pretest to posttest) spans about 10 days, nowhere near the one full academic term (roughly 3-4 months) required by the ERCT standard. There is no indication of a naturally short intervention combined with term-long follow-up tracking, which the standard would otherwise still require. Final sentence: Criterion T is not met because the interval from intervention start to outcome measurement was about ten days, far shorter than one academic term.
  • D

    Documented Control Group

    • The study compares three active instructional conditions rather than documenting a genuine untreated or business-as-usual control group with demographic detail.
    • "There were no pre-existing differences between the three experimental groups (PI, TI, and II) and any postintervention differences can be attributed to the effects of instruction." (p. 728)
    • Relevant Quotes: 1) "There were no pre-existing differences between the three experimental groups (PI, TI, and II) and any postintervention differences can be attributed to the effects of instruction." (p. 728) 2) "Phase 2 of the study aims to examine the effectiveness of pedagogical intervention using TI, PI, and II involving three experimental groups of primary 2 students 9-10 years old from St. Clement primary school in Hong Kong." (p. 722) 3) "In II, learners do not receive any explicit explanation of grammatical rules about English simple past. Learners are exposed to different passages with the use of English simple past. They first listen to the passages, read them and are then required to answer questions to check their understanding of the passages." (p. 723) Detailed Analysis: Criterion D requires a clearly documented control group, including demographics, baseline performance, and description of any treatment received. This study, however, does not include a genuine untreated or "business as usual" control condition; instead it compares three active pedagogical treatments (PI, TI, and II) against one another. Even the weakest arm (implicit instruction) still involves a full, purpose-built instructional package (15 activities, passages, comprehension questions) rather than standard classroom teaching or no intervention. While baseline pretest scores (M, SD) are reported per group in Tables 11-14, and group sizes are given, there is no demographic description (e.g., gender balance, socio-economic background) of any of the three groups, and none of the groups functions as a documented comparison/control condition in the sense intended by the standard. Final sentence: Criterion D is not met because the design compares three active treatment arms without a genuine, demographically documented control or business-as-usual condition.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred among individual students within a single primary school, not among multiple schools.
    • "Phase 2 of the study aims to examine the effectiveness of pedagogical intervention using TI, PI, and II involving three experimental groups of primary 2 students 9-10 years old from St. Clement primary school in Hong Kong." (p. 722)
    • Relevant Quotes: 1) "Phase 2 of the study aims to examine the effectiveness of pedagogical intervention using TI, PI, and II involving three experimental groups of primary 2 students 9-10 years old from St. Clement primary school in Hong Kong." (p. 722) 2) "There were 66 students assigned to one of the three groups randomly." (p. 722) Detailed Analysis: The entire Phase 2 experiment was conducted at a single school (St. Clement primary school), with individual students from that one school randomly assigned to one of three instructional groups. There is no school-level randomisation at all; the study does not involve multiple schools being assigned to different conditions. Criterion S explicitly requires randomisation among schools (or equivalent institutional units), which is entirely absent here. Final sentence: Criterion S is not met because the study was conducted at a single school with student-level, not school-level, randomisation.
  • I

    Independent Conduct

    • The same researcher who designed the study, the teaching materials, and trained the teachers also conducted the data collection and analysis, with no independent third-party evaluator.
    • "I sincerely thank the review panel and all the reviewers involved in making this study possible. I would also like to show my gratitude to SKH St Clemen's Primary School, specifically to Ms Ngai, the English teachers involved, and the students who joined the lessons." (p. 731, Acknowledgments)
    • Relevant Quotes: 1) "I sincerely thank the review panel and all the reviewers involved in making this study possible. I would also like to show my gratitude to SKH St Clemen's Primary School, specifically to Ms Ngai, the English teachers involved, and the students who joined the lessons." (p. 731, Acknowledgments) 2) "Three teachers were trained for the PI, TI, and II groups and they were convinced that they were doing something that has a high likelihood of success." (p. 722) 3) "This research was supported by Research Grants Council (RGC) of Hong Kong under General Research Fund (GRF) [grant number: PolyU 5404/13H]." (p. 731) Detailed Analysis: This is a single-author paper (Mable Chan). The Acknowledgments make clear that the school's teachers merely delivered lessons that the author designed and for which the author trained them ("Three teachers were trained for the PI, TI, and II groups"), while the author herself appears to have designed the intervention materials (Figures 1-3), the assessment tasks, and to have carried out the statistical analysis (via IBM SPSS). There is no mention of an independent evaluation team, external data collectors blind to condition, or any third-party organisation responsible for data collection or analysis. The school and teachers who implemented the lessons are not described as independent evaluators; they were trained by, and worked under, the same researcher who designed the study and is also the sole author who analysed and reported the results. Final sentence: Criterion I is not met because the same researcher designed, trained the implementers of, and analysed the intervention, with no independent third-party conduct of the evaluation.
  • Y

    Year Duration

    • Since the term-duration criterion (T) is not met, the stronger year-duration criterion is automatically not met; the whole study, in any case, spans only about ten days.
    • "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days. Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722)
    • Relevant Quotes: 1) "A pretest was given to the learners one week before the lessons and a posttest immediately after the lessons." (p. 723) 2) "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days. Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722) Detailed Analysis: Per the ERCT-specific instruction for criterion Y, if criterion T (Term Duration) is not met then Y is automatically not met. As established above, the interval between intervention start and outcome measurement was roughly ten days (one week pretest lead-in, three days of lessons, immediate posttest), which is nowhere close to even one term, let alone 75% of an academic year (~9-10 months). No delayed posttest was possible due to curriculum constraints. Final sentence: Criterion Y is not met, both because criterion T is not met and because the actual study duration (about ten days) is far shorter than 75% of an academic year.
  • B

    Balanced Control Group

    • All three groups received identical amounts of instructional time using materials explicitly balanced for difficulty and quantity, so the study isolates the effect of instructional approach rather than resource differences.
    • "A total of 15 activities were developed for each form of pedagogical intervention. They were balanced and comparable in terms of the difficulty of vocabulary, the number of activities, and verb tokens." (p. 723)
    • Relevant Quotes: 1) "A total of 15 activities were developed for each form of pedagogical intervention. They were balanced and comparable in terms of the difficulty of vocabulary, the number of activities, and verb tokens." (p. 723) 2) "They were taught for 1.5 hrs each day for a total of 4.5 hrs in three consecutive days." (p. 722), a duration applied identically to the PI, TI, and II groups. 3) "The experimental groups were taught by three different teachers to avoid the Hawthorne effect... Three teachers were trained for the PI, TI, and II groups and they were convinced that they were doing something that has a high likelihood of success." (p. 722) Detailed Analysis: This study is a three-arm comparison of instructional approaches (PI vs TI vs II) rather than a single treatment-vs-untreated-control design. Applying the Criterion B decision procedure: no group received extra time, materials, or teacher attention beyond what the others received (4.5 hours across 3 days, 15 activities each, one dedicated trained teacher per group), and the activities were explicitly described as balanced for difficulty and vocabulary load across all three conditions. Since no differential extra resource was given to only one arm, the criterion is trivially satisfied under the decision tree's first branch (no extra resources present relative to the other arms); the only systematic difference between groups is the pedagogical technique used within the same time and activity envelope, which is precisely what the study intends to test. Final sentence: Criterion B is met because all three instructional groups received equal, explicitly balanced amounts of time, activities, and dedicated teacher attention.

• Level 3 Criteria

  • R

    Reproduced

    • No evidence was found, in the paper or through external search, of an independent research team replicating this specific study; the only closely related publication located is by the same single author.
    • Relevant Quotes: The paper's own literature review discusses several processing-instruction studies on other structures and languages (e.g., Cadierno, 1995; Benati, 2001, 2009, 2010), but none of these is a replication of this specific Hong Kong primary-2 study on English simple past; they either predate it or examine different grammatical structures/languages. Internet search (Google Scholar-style search, Taylor & Francis, ScienceDirect, ResearchGate) located a closely related publication: "The Role of Classroom Input: Processing Instruction, Traditional Instruction, and Implicit Instruction in the Acquisition of the English Simple Past by Cantonese ESL Learners in Hong Kong" (Chan, M., System, 2019), which appears to report the same primary 2 PI/TI/II three-group study by the same single author (Mable Chan), not an independent replication by a different research team in a different context. No other independent replication of this exact study (same structure, same population, same PI/TI/II design, Hong Kong primary 2 Cantonese ESL learners) by a different research team was found. Detailed Analysis: Criterion R requires that this specific study's core experimental claim be independently replicated by a different research team in a different context and published in a peer-reviewed journal. While the broader processing-instruction literature is rich (VanPatten & Cadierno, 1993; Benati, several studies), these are conceptually related but not replications of this particular study's design, population (Cantonese ESL primary 2 students), and target structure (English simple past) as implemented by this single author. The one closely matching publication identified (System, 2019) shares the same author and appears to draw on the same dataset/design rather than constituting an independent replication. No independent replication of this specific study was identified. Final sentence: Criterion R is not met because no independent replication of this specific study by a different research team was found; the only closely related publication identified is by the same author.
  • A

    All-subject Exams

    • Only English grammar (simple past) was assessed, with no measurement of other core subjects, and criterion E (a prerequisite) is not met.
    • "The pretest and posttest consisted of five tasks (both interpretation and production tasks)—a sentence-level interpretation task, two discourse-level interpretation tasks, a fill-in-the-blanks task, and a sentence creation task—to determine the participants' ability of producing sentences using English simple past." (p. 723)
    • Relevant Quotes: 1) "The pretest and posttest consisted of five tasks (both interpretation and production tasks)—a sentence-level interpretation task, two discourse-level interpretation tasks, a fill-in-the-blanks task, and a sentence creation task—to determine the participants' ability of producing sentences using English simple past." (p. 723) 2) "It was also found that the three groups did not perform significantly different in reading (medium effect...) and listening tasks (small effect...), but better in sentence level... than in discourse level tasks." (p. 729) — all still measuring the same single grammatical target, not other subjects. Detailed Analysis: Per the criteria-specific instruction, since criterion E (Exam-based Assessment) is not met, criterion A cannot be met either. Independently, the study only ever assessed one narrow linguistic target (English simple past, via interpretation and production tasks); no other core subjects such as mathematics, science, or general language arts were measured. There is no stated rationale of a specialised vocational context that would justify this narrow scope under the exception for A. Final sentence: Criterion A is not met because criterion E is not met and because assessment was restricted to a single grammatical structure rather than covering the main subjects.
  • G

    Graduation Tracking

    • No follow-up beyond the immediate posttest was conducted or reported, and criterion Y (a prerequisite) is not met.
    • "Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722)
    • Relevant Quotes: 1) "Because of the curriculum constraint, no delayed posttests could be arranged." (p. 722) 2) "A pretest was given to the learners one week before the lessons and a posttest immediately after the lessons." (p. 723) Detailed Analysis: Per the criteria-specific instruction, since criterion Y (Year Duration) is not met, criterion G cannot be met either. Independently, the paper explicitly states that no delayed posttest could be arranged, meaning measurement stopped immediately after the three-day intervention. Internet search for follow-up publications by the same author tracking this cohort found no such paper; the only related later publication identified (Chan, M., System, 2019) reports the same Phase 2 dataset without any further follow-up or graduation tracking, and no other subsequent paper tracking this cohort through to graduation was found. Final sentence: Criterion G is not met because criterion Y is not met and because no follow-up beyond the immediate posttest, in this paper or any located subsequent publication, was found.
  • P

    Pre-Registered

    • The paper contains no statement of, or reference to, any pre-registered study protocol, and no matching registration was found in registry searches.
    • Relevant Quotes: No quotes referencing a pre-registration platform, registry ID, or pre-specified analysis plan were found anywhere in the Method, Results, or Discussion sections of the paper. Detailed Analysis: Criterion P requires a quoted reference to a pre-registration (e.g., on a registry platform) with a date prior to data collection. The Method section describes ethics-adjacent details such as consent forms for the Phase 1 questionnaire ("They all signed a consent form agreeing to participate in this study") but makes no mention of pre-registering hypotheses, methods, or analysis plans for either Phase 1 or Phase 2 before data collection began. An internet search for a pre-registration of this study (e.g., OSF, AEA RCT Registry, ClinicalTrials.gov-style registries) found no matching record for this author or study. Final sentence: Criterion P is not met because no evidence of a pre-registered protocol is present in the paper or in registry searches.