Using Task-based Instruction to Develop EFL Learners' Pragmatic Competence: The Case of Oral Fluency

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was carried out at the individual-student level from a pool of candidates, not at the class or school level, and no personal-tutoring exception applies.
    • "They were randomly assigned into two equal groups (i.e., each 32 participants) of a control (CG) receiving a conventional approach (i.e., Presentation, practice, production, PPP), and an experimental (EG) acquiring pragmatic competence of oral fluency." (p. 138)
    • Relevant Quotes: 1) "They were randomly assigned into two equal groups (i.e., each 32 participants) of a control (CG) receiving a conventional approach (i.e., Presentation, practice, production, PPP), and an experimental (EG) acquiring pragmatic competence of oral fluency." (p. 138) 2) "The participants in the present study were 64 students recruited from among 118 students majoring in teaching EFL." (p. 141) 3) "To select the research sample, the Oxford Placement Test (OPT) was used and 64 learners who achieved a band score between 48 and 54 were divided into EG (N=32) and CG (N=32)." (p. 141) Detailed Analysis: The paper describes a pool of 118 eligible students from a single English-language teaching program, from which 64 individually-screened students (via OPT band score) were randomly allocated to EG or CG. This is randomisation of individual students drawn from one population, not randomisation of pre-existing intact classes or schools. There is no mention of the intervention being a one-to-one/personal tutoring format that would trigger the Exception for individual-level randomisation; instead the intervention was delivered to groups of 32 students each via pair/group classroom activities. Criterion C is not met because randomisation occurred at the student level within a single institute, without a valid tutoring exception.
  • E

    Exam-based Assessment

    • The pragmatic-competence outcome was measured with a researcher-made observation checklist, not a standardised exam; the only standardised instrument (OPT) was used solely for proficiency screening.
    • "To assess pragmatic oral fluency, the measures taken in the model were employed by researcher-made observation checklist including the range of 1 to 20 points." (p. 141)
    • Relevant Quotes: 1) "The OPT was just used to determine the proficiency level of the participants rather than to measure their pragmatic competence." (p. 141) 2) "To assess pragmatic oral fluency, the measures taken in the model were employed by researcher-made observation checklist including the range of 1 to 20 points." (p. 141) 3) "The classroom observation checklist was also used to prepare reports on the learners in both EG and CG. It was designed based on Masuram and NaginiSripada (2020) and then it was validated in a pilot study." (p. 141) Detailed Analysis: The only standardised instrument used, the Oxford Placement Test (OPT), is explicitly stated to serve only as a proficiency-screening tool for sample selection, not as the outcome measure. The actual dependent variable (pragmatic oral fluency) was scored using a researcher-made observation checklist adapted from another study and validated only in the authors' own pilot study, not a widely recognised, independently standardised exam. Criterion E is not met because the primary outcome assessment is a custom, researcher-made checklist rather than a standardised exam-based assessment.
  • T

    Term Duration

    • The intervention and post-test spanned a whole semester (14 sessions), matching the ERCT definition of at least one academic term.
    • "Then they participated in intervention sessions of text-based exercises in the CG and TBI in the EG for a whole semester." (p. 138)
    • Relevant Quotes: 1) "Then they participated in intervention sessions of text-based exercises in the CG and TBI in the EG for a whole semester." (p. 138) 2) "In the intervention phase, EG participated in TBI and received meaningful and communicative tasks (i.e., pre-task, task, post-task) within 14 treatment sessions. Each session lasted for 90 minutes..." (p. 142) 3) "Limitations of the study included the short time of the research period, conducting only one semester (14 sessions)." (p. 145) Detailed Analysis: The abstract and limitations section both confirm the intervention ran across "a whole semester" (14 sessions), with the post-test administered at the end of that semester. A semester meets the ERCT definition of a term (approximately 3-4 months), satisfying the requirement that outcomes be measured at least one full academic term after the intervention begins. Criterion T is met because the intervention-to-measurement interval spans a full academic semester.
  • D

    Documented Control Group

    • The control group's size, instructional content, and baseline pretest scores are described in detail, allowing comparison with the experimental group.
    • "CG received a conventional conversation class of the PPP model by listening to the textbook conversations and doing the exercises orally." (p. 142)
    • Relevant Quotes: 1) "CG received a conventional conversation class of the PPP model by listening to the textbook conversations and doing the exercises orally. Participants attended conversation class that adhered to the presentation phase, during which the instructor introduced the target conversational structures and expressions, frequently utilizing brief dialogues from the designated textbook." (p. 142) 2) "14 units of the textbook were covered in both groups." (p. 142) 3) Table 1 (Normality Test): Pretest, CG, N=32, Mean=10.68, SD=1.89, compared with EG, N=32, Mean=10.28, SD=1.95. (p. 143) Detailed Analysis: The methods section gives a detailed narrative of what the control group experienced (the PPP presentation/practice/production sequence, textbook- and CD-based, teacher-fronted instruction), its size (N=32), and its pretest baseline scores reported separately from the experimental group in Table 1. This level of detail allows a reader to assess comparability of the two groups at baseline and to understand what "business as usual" instruction the CG received. Criterion D is met because the control group's composition, treatment, and baseline performance are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Only a single institute's student population was involved; randomisation was at the individual level, not across schools.
    • "The participants in the present study were 64 students recruited from among 118 students majoring in teaching EFL." (p. 141)
    • Relevant Quotes: 1) "The participants in the present study were 64 students recruited from among 118 students majoring in teaching EFL." (p. 141) 2) "To select the research sample, the Oxford Placement Test (OPT) was used and 64 learners who achieved a band score between 48 and 54 were divided into EG (N=32) and CG (N=32)." (p. 141) Detailed Analysis: There is no mention anywhere in the paper of multiple schools/institutions being randomised; the entire study was conducted within one English-language teaching program, with individual students randomly split into two groups. This does not meet the stronger school-level randomisation requirement, nor does it satisfy the weaker class-level requirement (see criterion C). Criterion S is not met because randomisation occurred at the individual-student level within a single institute rather than across schools.
  • I

    Independent Conduct

    • No independent or third-party evaluators are mentioned; the same author team appears responsible for the whole study.
    • "All authors have contributed equally to prepare the paper." (p. 146)
    • Relevant Quotes: 1) "Authors' Contributions: All authors have contributed equally to prepare the paper." (p. 146) 2) No statement anywhere in the Methodology, Data Collection Procedure, or other sections indicates that instruction, data collection, or analysis were performed by an external/independent evaluation team distinct from the authors. Detailed Analysis: The paper gives no indication of any third-party or external evaluation team being involved in delivering the intervention, collecting the observation-checklist data, or analysing results. The two named authors appear to have jointly handled all aspects of the study ("All authors have contributed equally to prepare the paper"), with no disclosure of independent oversight or blinded external raters. Criterion I is not met because there is no evidence of independent conduct separate from the study's designers.
  • Y

    Year Duration

    • The study lasted only one semester (14 sessions), far short of 75% of a full academic year.
    • "Limitations of the study included the short time of the research period, conducting only one semester (14 sessions)." (p. 145)
    • Relevant Quotes: 1) "Then they participated in intervention sessions of text-based exercises in the CG and TBI in the EG for a whole semester." (p. 138) 2) "Limitations of the study included the short time of the research period, conducting only one semester (14 sessions). The future research may implement task-based learning over a longer period, resulting in other findings." (p. 145) Detailed Analysis: The authors themselves flag the one-semester duration as a limitation of the study, contrasting it with a hypothetical longer-term design. A single semester (roughly 14 sessions) falls well short of the ERCT requirement of at least 75% of a full academic year (~9-10 months). Criterion Y is not met because the intervention and follow-up period covered only one semester, not close to a full academic year.
  • B

    Balanced Control Group

    • Both groups received the same 14 textbook units/sessions; the only difference was pedagogical approach (TBI vs. PPP), which is the treatment variable itself.
    • "14 units of the textbook were covered in both groups." (p. 142)
    • Relevant Quotes: 1) "In the intervention phase, EG participated in TBI and received meaningful and communicative tasks (i.e., pre-task, task, post-task) within 14 treatment sessions. Each session lasted for 90 minutes..." (p. 142) 2) "14 units of the textbook were covered in both groups." (p. 142) 3) "CG received a conventional conversation class of the PPP model by listening to the textbook conversations and doing the exercises orally... Furthermore, learners listened to the CD or the teacher and then answered the textbook exercises." (p. 142) Detailed Analysis: Applying the criterion B decision procedure: extra resources are not disproportionately present in one arm (EXTRA_RESOURCES_PRESENT is effectively false) because both EG and CG covered the same "14 units of the textbook" within the same one-semester window, implying matched session count and instructional time (EG's sessions are explicitly 90 minutes each; CG's parallel structure covers the same 14 units over the same period, with no quote indicating CG received less contact time or fewer materials). The variable under test is the teaching methodology itself (task-based instruction vs. presentation-practice-production), not an added resource (extra time, budget, or materials) given only to one group. Since no additional, non-integral resource was identified for either arm, the RESOURCES_ARE_TREATMENT / CONTROL_MATCHES_RESOURCES branches are not triggered, and the decision falls at step 1 of the decision tree ("no extra resources present" -> met). Criterion B is met because both groups received matched instructional time and textbook content, with the pedagogical method itself being the tested variable and no imbalance in time, budget, or materials identified.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study is reported in the paper, and an internet search found no subsequent replication by another research team.
    • Relevant Quotes: No quotes in the paper reference a prior or subsequent independent replication of this specific TBI-vs-PPP pragmatic oral fluency study by a different research team. Detailed Analysis: The paper is presented as an original, standalone study. While it cites related literature on TBI and pragmatic competence (e.g., Vahid Dastjerdi & Rezvani, 2010; Omar & Razi, 2022), none of these are replications of this exact study design, sample, or measures; they are separate studies on related but distinct interventions. Internet-backed re-check: this article was published online on 30 September 2025 (DOI 10.57647/jals.2025.0501.12). A citation-count lookup via OpenAlex on the paper's DOI returned "cited_by_count": 0, and targeted Google Scholar and Semantic Scholar searches for independent replications of this specific TBI-vs-PPP pragmatic-oral-fluency design by other research teams returned no matching results. Given the very short time elapsed since publication, no independent replication currently exists in any indexed source. Criterion R is not met because no independent replication of this specific study was found in the text or via internet search.
  • A

    All-subject Exams

    • Criterion E (standardised exam-based assessment) is not met, so this stronger criterion is automatically not met.
    • Relevant Quotes: 1) "To assess pragmatic oral fluency, the measures taken in the model were employed by researcher-made observation checklist including the range of 1 to 20 points." (p. 141) Detailed Analysis: Per the ERCT specification, Criterion A requires Criterion E to be met as a prerequisite. Since the study's only outcome measure is a custom, researcher-made observation checklist of pragmatic oral fluency (not a standardised exam), and no other subjects or standardised exams were assessed, Criterion A automatically fails. Criterion A is not met because Criterion E is not met and no standardised, all-subject assessment was used.
  • G

    Graduation Tracking

    • No follow-up beyond the immediate post-test was conducted, Criterion Y is not met, and an internet search found no subsequent graduation-tracking publication by the same authors.
    • "The other limitations address the lack of long-term assessment regarding TBI effectiveness on learners' oral fluency and pragmatic competence, since it is not conducted in the study." (p. 145)
    • Relevant Quotes: 1) "In the final phase, an oral posttest was utilized the same as the pretest." (p. 142) 2) "The other limitations address the lack of long-term assessment regarding TBI effectiveness on learners' oral fluency and pragmatic competence, since it is not conducted in the study." (p. 145) 3) "Moreover, this study assessed the short-term effects of TBI on learners' oral fluency." (p. 145) Detailed Analysis: Data collection ended with the immediate post-test at the close of the one-semester intervention. The authors explicitly acknowledge the lack of long-term or follow-up assessment as a limitation, with no mention of tracking participants toward graduation. Internet-backed re-check: a citation-count lookup via OpenAlex for this paper's DOI returned zero citing works, and searches for follow-up publications by Gorjian and/or Mir tracking this same 64-student cohort toward graduation returned no results. No such follow-up paper could be located. This criterion also fails automatically because Criterion Y is not met. Criterion G is not met because no follow-up tracking toward graduation was conducted or reported, and no subsequent tracking publication was found via internet search.
  • P

    Pre-Registered

    • No mention of a pre-registered protocol, registry platform, or registration date is found anywhere in the paper, and no internet search of trial registries located a pre-registration record for this study.
    • Relevant Quotes: No quotes referencing pre-registration, a trial registry (e.g., ClinicalTrials.gov, ISRCTN, OSF), or a pre-specified analysis plan were found in the Methodology, Data Collection Procedure, or Availability of Data and Materials sections. Detailed Analysis: The paper contains no statement about the study protocol, hypotheses, or analysis plan being registered before data collection began. The "Availability of Data and Materials" statement only mentions that data are available upon reasonable request, with no reference to pre-registration. Internet-backed re-check: searches for a pre-registration record under the authors' names (Bahman Gorjian, Farzaneh Mir) or this study's title/topic on common registries and via web search found no matching pre-registration entry. Criterion P is not met because there is no evidence of a pre-registered protocol in the paper or in any external registry search.