The Impact of Metacognitive Strategies on Jordanian EFL Learners' Writing Performance

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Two pre-existing intact classes were assigned to conditions with no described randomisation, so a class-level RCT is not documented.
    • "Since the students had been placed in two different interactive classes in advance by their educational program, one class was assigned as the control and the other as the experimental group." (p. 330)
    • Relevant Quotes: 1) "Forty four students were randomly selected from secondary school level to partake in experimental control of the study." (p. 328, Abstract) 2) "Since the students had been placed in two different interactive classes in advance by their educational program, one class was assigned as the control and the other as the experimental group." (p. 330) 3) "This research involved 44 EFL students who were in the last year of secondary school of Al-Mazar Irbid in Jordan." (p. 330) Detailed Analysis: Criterion C requires a clearly described randomisation at the class level (or stronger). The abstract mentions that students were "randomly selected", but this refers to sampling of participants, not to random assignment to conditions. The methodology explicitly states that the two classes already existed ("had been placed ... in advance by their educational program") and that "one class was assigned as the control and the other as the experimental group" with no statement that this assignment of the two intact classes to conditions was done randomly. With only two pre-existing classes and no description of a random allocation procedure, the study is effectively a quasi-experiment on intact groups rather than a properly implemented class-level RCT. The intervention is classroom-based writing instruction, not one-to-one tutoring, so the tutoring exception does not apply. I re-read the Methodology section in full and confirmed no other passage describes a random or chance-based procedure for allocating the two classes to condition, so this verdict is retained. Criterion C is not met because assignment of the two intact classes to experimental and control conditions is not described as random, so class-level randomisation is not documented.
  • E

    Exam-based Assessment

    • Outcomes were measured with a custom researcher-made writing test scored by the researchers, not a recognised standardised exam.
    • "The test question was given from the course materials that assumed as the source book of their current study." (p. 330)
    • Relevant Quotes: 1) "In this study, there were various instruments used which included, pre-test, immediate post-test, and delay test. The test question was given from the course materials that assumed as the source book of their current study." (p. 330) 2) "The students were tasked to write on a topic with about 150 words in an essay format." (p. 330) 3) "Their language proficiency was determined by holistic scoring by two researchers." (p. 330) 4) "All participants from the experimental and control groups were expected to take one pre-test and one post-writing a test to assess whether there was an improvement in writing performances over several weeks of training in the research. The test provided data for measuring participants' writing performance." (p. 331) Detailed Analysis: Criterion E requires the use of widely recognised standardised exams rather than assessments created for the study. Here the outcome measure was a writing task devised by the researchers, with topics drawn from the students' course materials and scored holistically by two of the researchers themselves. No named national or standardised examination (e.g., a national curriculum exam, IELTS, TOEFL) is mentioned anywhere in the paper, and no validity or reliability evidence for the test is reported. This is precisely the custom, study-specific assessment the criterion warns against. Criterion E is not met because outcomes were measured with a researcher-made writing test based on course materials rather than a recognised standardised exam.
  • T

    Term Duration

    • Outcomes were measured 12 weeks after the intervention began, which approximately equals one full academic term.
    • "Also, both groups receive the delay post-test at the end of instruction program week twelve." (p. 330)
    • Relevant Quotes: 1) "The class met for about two hours every week for 12 weeks." (p. 330) 2) "After five weeks of instruction, both groups were immediately post-tested. Also, both groups receive the delay post-test at the end of instruction program week twelve." (p. 330) 3) "After five weeks of instruction, both groups were post-tested and at the end of program which lasted for twelve weeks, the students carried out another post-test." (p. 328, Abstract) 4) "The instruction was for a period of twelve weeks." (p. 335) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. The intervention began at week 1 and the final (delayed) post-test was administered at week 12, so the interval from intervention start to the primary final measurement is 12 weeks, roughly 3 months. Twelve weeks corresponds to the length of a standard academic term or quarter in many systems and sits at the lower bound of the "approximately 3-4 months" definition in the standard. The measurement was not immediate after a brief intervention; tracking spanned the full 12-week program. This is a borderline case, but the quoted interval reasonably constitutes one academic term. Criterion T is met because the delayed post-test occurred 12 weeks (about one full academic term) after the intervention began.
  • D

    Documented Control Group

    • The control group's size, population, routine instruction, and baseline pre-test scores are documented in the text and Table 1.
    • "The experimental group (EG) received metacognitive strategies-based writing instruction whereas the control group (CG) received only the routine writing instruction (Product Approach)." (p. 330)
    • Relevant Quotes: 1) "This research involved 44 EFL students who were in the last year of secondary school of Al-Mazar Irbid in Jordan. They were at the secondary school level. Their language proficiency was determined by holistic scoring by two researchers." (p. 330) 2) "The experimental group (EG) received metacognitive strategies-based writing instruction whereas the control group (CG) received only the routine writing instruction (Product Approach)." (p. 330) 3) "Pre-test +MST (Exp.) 22 12.50 1.045 .158 / -MST (Contr.) 22 12.10 1.649 .249" (Table 1, p. 331) 4) "A pretest was conducted before the experiment to ascertain that the writing abilities of the two classes were moving at the same speed." (p. 331) 5) "In order to establish the homogeneity of the two groups in terms of their overall performance in writing, the Mann-Whitney U test, was carried out." (p. 332) Detailed Analysis: Criterion D requires clear documentation of the control group: who they are, their size, baseline performance, and what treatment they received. The paper identifies the control group as 22 final-year secondary school EFL students in Al-Mazar, Irbid, Jordan, states explicitly that they received only routine Product Approach writing instruction, and reports their baseline pre-test mean, standard deviation and standard error in Table 1, with a statistical homogeneity check against the experimental group. Demographic detail is thin (no age or gender breakdown per group), and the reported pre-test control mean is inconsistent between Table 1 (12.10) and the narrative (11.55), which weakens the documentation somewhat. Still, the size, condition, and baseline performance of the control group are documented at a level comparable to what the standard requires. Criterion D is met because the control group's size (n=22), population, business-as-usual instruction, and baseline pre-test performance are documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Only two intact classes in one setting were used; there was no randomisation of schools.
    • "Since the students had been placed in two different interactive classes in advance by their educational program, one class was assigned as the control and the other as the experimental group." (p. 330)
    • Relevant Quotes: 1) "This research involved 44 EFL students who were in the last year of secondary school of Al-Mazar Irbid in Jordan." (p. 330) 2) "Since the students had been placed in two different interactive classes in advance by their educational program, one class was assigned as the control and the other as the experimental group." (p. 330) Detailed Analysis: Criterion S requires randomisation among schools (or equivalent implementing institutions). This study took place with two classes drawn from the secondary school level in a single locality (Al-Mazar, Irbid), and the unit of assignment was the pre-existing class, not the school. No multiple schools were recruited and no school-level randomisation is described anywhere in the paper. Criterion S is not met because assignment involved two intact classes within a single setting, with no school-level randomisation.
  • I

    Independent Conduct

    • The same researchers designed, taught, scored, and analysed the intervention with no independent evaluation team.
    • "During this time, the researcher employed meta-cognitive learning strategies and taught the participants in the experimental group how to utilize metacognitive strategies in their writing skill." (p. 335)
    • Relevant Quotes: 1) "The researcher made use of the intervention program based on CALLA model of teaching in classroom." (p. 328, Abstract) 2) "In the phase of preparation, the researchers first assist the students to associate what they know about the contents and strategies..." (p. 330) 3) "The researchers first gave out a list of the metacognitive strategies in writing including self-preparation, self-monitoring and self-assessment." (p. 331) 4) "Their language proficiency was determined by holistic scoring by two researchers." (p. 330) 5) "During this time, the researcher employed meta-cognitive learning strategies and taught the participants in the experimental group how to utilize metacognitive strategies in their writing skill." (p. 335) Detailed Analysis: Criterion I requires that the study be conducted independently of the intervention designers, or at least with documented third-party oversight of data collection and analysis. In this paper the same research team designed the intervention program, delivered the instruction in the classroom, created the writing tests, scored them holistically themselves, and analysed the results. There is no mention of any external evaluator, independent test administrators, blinded scorers, or third-party oversight of any part of the study. Criterion I is not met because the authors designed, delivered, scored, and analysed the intervention themselves with no independent oversight.
  • Y

    Year Duration

    • The study lasted only 12 weeks from start to final measurement, far below 75% of an academic year.
    • "The instruction was for a period of twelve weeks." (p. 335)
    • Relevant Quotes: 1) "The class met for about two hours every week for 12 weeks." (p. 330) 2) "The instruction was for a period of twelve weeks." (p. 335) 3) "Also, both groups receive the delay post-test at the end of instruction program week twelve." (p. 330) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of a full academic year (roughly 7+ months) after the intervention begins. The entire study, from intervention start to the final delayed post-test, lasted 12 weeks. This is far below the year-duration threshold. Additionally, per the standard's dependency, criterion T (term duration) is not met, which also rules out criterion Y. Criterion Y is not met because the tracking interval was only 12 weeks, far short of 75% of an academic year.
  • B

    Balanced Control Group

    • Both groups had the same class time and received writing instruction, differing only in the teaching method being tested, which is integral to the intervention.
    • "The class met for about two hours every week for 12 weeks." (p. 330)
    • Relevant Quotes: 1) "The class met for about two hours every week for 12 weeks." (p. 330) 2) "The experimental group (EG) received metacognitive strategies-based writing instruction whereas the control group (CG) received only the routine writing instruction (Product Approach)." (p. 330) 3) "Then, the instruction on the metacognitive learning strategies for the experimental group was considered in 40 minutes." (p. 330) 4) "All participants from the experimental and control groups were expected to take one pre-test and one post-writing a test to assess whether there was an improvement in writing performances over several weeks of training in the research." (p. 331) Detailed Analysis (Decision-Tree Application): Criterion B requires that the control group receive balanced time and resources unless the extra resources are themselves the explicit treatment variable. Both intact classes followed the same overall weekly schedule ("about two hours every week for 12 weeks") and both received writing instruction during regular class time from the same research team; the classes differ only in teaching method: the experimental class received CALLA-based metacognitive strategies instruction (including a 40-minute segment devoted to explicit strategy instruction) while the control class received the routine Product Approach writing instruction. No quote indicates the experimental class received additional total class time, extra sessions, additional materials, or extra budget beyond its regular weekly session; the 40-minute segment is described as occurring within that same class period, not as time added on top of it. Applying the criterion B decision tree: EXTRA_RESOURCES_PRESENT is not established by any quote (no stated increase in total class time or budget for the experimental group relative to control), so the criterion is satisfied at the first branch (met without extra resources). Even under the more conservative reading that the 40-minute strategy segment is an additional resource, it is integral to the intervention being tested (RESOURCES_ARE_TREATMENT): the metacognitive-strategy method delivered in that time is itself the treatment variable being compared against the routine Product Approach, so the criterion is still met under that branch as well. Criterion B is met because both groups received writing instruction in the same weekly class time from the same researchers, with the metacognitive method itself being the only described difference and, if it is regarded as an added resource at all, an integral part of the tested intervention.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study is reported in the paper, and an internet citation search found none published since.
    • Relevant Quotes: 1) "Furthermore, the findings of this study agree with the results obtained by Javed (2013) in which the same CALLA model was adopted." (p. 334) 2) "Moreover, a study close to the present one was conducted by Aliweh (2011) who investigated the effects of a proposed strategy-based writing model on Saudi EFL students' writing skills." (p. 335) Internet Search for Independent Replication: A search of the Semantic Scholar citation record for this paper's DOI (10.5539/ijel.v8n6p328) and general web searches for the authors' names combined with the study topic were conducted. The citing works identified (Roslaini & Dwiyanti 2023; Chen, Gong, Liu & Cheng 2023; Ebrahimi, Izadpanah & Namaziandost 2021; Fitrianti & Susanti 2021; Rivero Galeano, Gomez Salgado & Lorduy Arellano 2020; Kamhieh 2020) discuss metacognitive writing strategies generally or cite this paper as background; none of them attempts to independently reproduce this specific Jordanian secondary-school, CALLA-based writing intervention with a different research team in a different context and report a comparison of results in a peer-reviewed journal. Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different team, in a different context, and published in a peer-reviewed journal. The related studies the authors discuss (Javed 2013, Hartman 2015, Aliweh 2011, Johnson 1991) all predate this 2018 paper and are prior similar investigations, not replications of this trial; they differ in population, design, and measures. The internet search of post-2018 citing literature likewise identified no independent replication of this specific study. Criterion R is not met because no independent published replication of this specific study exists; cited similar studies are prior work, and a citation-based search of post-2018 literature found no replication attempt.
  • A

    All-subject Exams

    • Only EFL writing was assessed with a custom test, so neither the standardised-exam prerequisite nor all-subject coverage is satisfied.
    • "The test provided data for measuring participants' writing performance." (p. 331)
    • Relevant Quotes: 1) "The test provided data for measuring participants' writing performance." (p. 331) 2) "The results showed that there was a positive effect in the experimental group's writing performance." (p. 328, Abstract) Detailed Analysis: Criterion A requires standardised exam-based assessment of all main school subjects, and it presupposes criterion E. Here criterion E is not met (the outcome measure is a custom researcher-made writing test), which by itself means criterion A cannot be met. Moreover, the study measured only English writing performance; no other subject (mathematics, science, Arabic, etc.) was assessed, and no justification for a specialised-scope exception is offered. Criterion A is not met because criterion E fails and only a single outcome domain (EFL writing) was measured.
  • G

    Graduation Tracking

    • Tracking ended at the week-12 delayed post-test with no follow-up to graduation, and none was found via internet search.
    • "Also, both groups receive the delay post-test at the end of instruction program week twelve." (p. 330)
    • Relevant Quotes: 1) "Also, both groups receive the delay post-test at the end of instruction program week twelve." (p. 330) 2) "After the posttest, the results indicated that the instruction of meta-cognitive learning strategies improved the learners' writing skill." (p. 335) Internet Search for Follow-Up Publications: A search of the Semantic Scholar citation record for this paper's DOI (10.5539/ijel.v8n6p328) and general web searches for the authors' names combined with the study topic were conducted, looking specifically for a follow-up publication by Al-Jarrah, Mansor, or Rashid tracking the same 44-student Al-Mazar Irbid cohort further. The citing papers found (Roslaini & Dwiyanti 2023; Chen, Gong, Liu & Cheng 2023; Ebrahimi, Izadpanah & Namaziandost 2021; Fitrianti & Susanti 2021; Rivero Galeano, Gomez Salgado & Lorduy Arellano 2020; Kamhieh 2020) are unrelated studies by other authors that merely cite this paper as background literature; none is a follow-up study by the original authors, and no subsequent publication tracking this cohort to graduation was found. Detailed Analysis: Criterion G requires tracking participants until graduation from their educational stage, and it presupposes criterion Y, which is not met here. Measurement ended with the delayed post-test at week 12 of the program. Although the participants were in their final year of secondary school, the paper reports no tracking to end-of-year or graduation outcomes, and internet search found no planned or published follow-up study of this cohort. Criterion G is not met because measurement stopped at the 12-week delayed post-test, criterion Y is not met, and no graduation-tracking follow-up publication could be located.
  • P

    Pre-Registered

    • The paper contains no reference to pre-registration or any trial registry, and none was found via internet search.
    • Relevant Quotes: No quotes mentioning pre-registration, a trial registry, a registration ID, or a published protocol appear anywhere in the paper. The article history states only: "Received: April 16, 2018 Accepted: June 27, 2018 Online Published: November 26, 2018" (p. 328). Internet Search for a Pre-Registration Record: A search was conducted for a pre-registration record under the authors' names and the study title/topic. No entry was found on any public registry (e.g., OSF Registries, ClinicalTrials .gov, AEA RCT Registry) associated with Al-Jarrah, Mansor, or Rashid's Jordanian EFL metacognitive writing study. Detailed Analysis: Criterion P requires that the full study protocol, including hypotheses and planned analyses, be registered on a public registry before data collection began. The paper contains no reference to any registry (e.g., ClinicalTrials.gov, OSF, AEA), no registration number, and no protocol publication. Internet search corroborates the absence of any public pre-registration record for this study. Criterion P is not met because there is no mention of any pre-registered protocol or registry entry in the paper, and no such record could be located online.