Reading Comprehension: Blog Exposure in the English as a Foreign Language (EFL) Context of Iran

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the individual student level within a single institute, not at the class or school level, and no personal-tutoring exception applies.
    • "After administrating the CPT, 40 students who were randomly and equally assigned to the experimental and control groups (20 students in each group) were selected as the sample of this study." (p. 60)
    • Relevant Quotes: 1) "After administrating the CPT, 40 students who were randomly and equally assigned to the experimental and control groups (20 students in each group) were selected as the sample of this study." (p. 60) 2) "The participants of this study were intermediate female EFL students at a private English language institute located in Kerman, Iran." (p. 60) 3) "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61) Detailed Analysis: The paper explicitly states that individual students, not classes or schools, were the unit of randomisation: 40 students were randomly and equally assigned to the experimental and control groups. There is no description of multiple pre-existing classes being randomised as whole units, nor of school-level assignment. The intervention (group reading-comprehension practice via blog vs. paper worksheets) is a classroom-style activity administered to groups of students, not personal one-to-one tutoring, so the ERCT tutoring exception does not apply. Because randomisation occurred below the class level within one institute, contamination between treatment and control students cannot be ruled out. Criterion C is not met because randomisation was performed at the individual student level within one institute and no tutoring exception applies.
  • E

    Exam-based Assessment

    • The outcome measure was a researcher/institute-designed reading test tied to the course syllabus, not a widely recognised standardised exam.
    • "The reading comprehension test contained thirty five multiple-choice and open ended questions." (p. 60)
    • Relevant Quotes: 1) "Two sets of reading comprehension assessment scores were used as measures in this study." (p. 60) 2) "The reading comprehension test contained thirty five multiple-choice and open ended questions." (p. 60) 3) "The content of the tests focused on the topics covered in the syllabus at this level and did not include new items." (p. 60) 4) "To consider the internal consistency reliability ... split-half reliability as a subtype of internal consistency reliability was used ... The reliability of the test was (0.89). The researcher also consulted experts and specialists in English language and methodology for referring the validity and the reliability of the study tools." (p. 60-61) 5) "A Cambridge placement test (CPT) was used to have almost homogenous groups." (p. 60) Detailed Analysis: The Cambridge Placement Test (CPT) was used only to form comparable groups before the study began, not as the pre/post outcome measure. The actual reading comprehension pre-test and post-test (35 multiple-choice and open-ended items) were explicitly tied to "the topics covered in the syllabus at this level" of the institute's own course book, indicating a custom, in-house instrument rather than a widely recognised standardised exam. The authors themselves computed split-half reliability for this custom instrument and consulted local experts for validity, which is characteristic of a researcher-built test rather than an externally validated, standardised assessment. Criterion E is not met because the outcome measure was a custom test aligned to the institute's own syllabus rather than a recognised standardised exam.
  • T

    Term Duration

    • The intervention and outcome measurement spanned a full academic semester (one term), satisfying the term-duration requirement.
    • "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61)
    • Relevant Quotes: 1) "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61) 2) "After 22 sessions, a post-test was conducted to measure the degree of improvement in learners' reading comprehension at the end of the implementation period." (p. 61) 3) "They were all native speakers of Persian and had previously attended 5-10 terms of general English instruction." (p. 60) Detailed Analysis: The paper states the intervention ran across "one semester" comprising 22 sessions, with the post-test (the primary outcome measure) administered at the end of this period, i.e., at the end of an intervention that itself spans a full term. A semester is a standard equivalent of an academic term (roughly 3-4 months). Since the outcome was measured at the conclusion of this full-semester intervention, the interval from intervention start to outcome measurement satisfies the "at least one term" requirement. Criterion T is met because the study tracked outcomes across a full academic semester (one term) from intervention start to post-test.
  • D

    Documented Control Group

    • The control group's size, baseline scores, and specific business-as-usual activity (paper worksheets) are clearly documented.
    • "During this period, the control group was given paper-based reading comprehension worksheet homework. Each worksheet corresponded to the topic and language of one unit of the course book." (p. 61)
    • Relevant Quotes: 1) "During this period, the control group was given paper-based reading comprehension worksheet homework. Each worksheet corresponded to the topic and language of one unit of the course book." (p. 61) 2) "Both groups were given three days to complete the reading task homework." (p. 61) 3) "Control 20 13.55 2.58" (Table 1, pre-test descriptive statistics for the control group, p. 61) 4) "The participants of this study were intermediate female EFL students at a private English language institute located in Kerman, Iran. They ranged in age from 15 to 25 ... They were all native speakers of Persian and had previously attended 5-10 terms of general English instruction." (p. 60) Detailed Analysis: The paper reports the control group's exact size (n=20), its pre-test baseline mean and standard deviation (M=13.55, SD=2.58), and a clear description of what the control group received during the study period (paper-based reading worksheets matched in topic and timing to the experimental group's blog activities). Sample-wide demographic information (age range, gender, native language, prior English instruction) is also given, and both groups were drawn from a pool verified as homogeneous via the CPT. This level of detail permits assessment of whether the control group was comparable at baseline and clarifies what conditions it experienced. Criterion D is met because the control group's size, baseline scores, and specific business-as-usual activity are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The trial was conducted with individual students inside a single institute; no school-level randomisation is described.
    • "The participants of this study were intermediate female EFL students at a private English language institute located in Kerman, Iran." (p. 60)
    • Relevant Quotes: 1) "The participants of this study were intermediate female EFL students at a private English language institute located in Kerman, Iran." (p. 60) 2) "After administrating the CPT, 40 students who were randomly and equally assigned to the experimental and control groups (20 students in each group) were selected as the sample of this study." (p. 60) Detailed Analysis: The ERCT S criterion requires randomisation among whole schools/institutions, not individual students within one institution. Here, all 40 participants were drawn from a single private language institute and randomised at the student level. There is no mention of multiple institutes or schools being compared or randomised as units. Criterion S is not met because randomisation occurred among individual students within a single institute rather than among schools.
  • I

    Independent Conduct

    • No statement indicates that an external or independent team conducted data collection or analysis separate from the researchers who designed the study.
    • Relevant Quotes: 1) "Mehry Haddad Narafshan (Corresponding author), Department of Foreign Languages, Kerman Branch, Islamic Azad University, Kerman, Iran." (author affiliation, p. 56) 2) "The researcher also consulted experts and specialists in English language and methodology for referring the validity and the reliability of the study tools." (p. 60-61) Detailed Analysis: The paper contains no statement describing an external evaluation team, independent data collectors, or blinded assessors. The only mention of any outside involvement is that "experts and specialists" were consulted for instrument validity/reliability, which does not amount to independent conduct of data collection or analysis. The design, implementation (including the demonstration of blog use) and testing all appear to have been carried out by the authors/researcher themselves. Criterion I is not met because there is no evidence of an independent, third-party team conducting the study's data collection or analysis.
  • Y

    Year Duration

    • The study tracked outcomes for one semester only, well short of the 75%-of-academic-year threshold required by criterion Y.
    • "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61)
    • Relevant Quotes: 1) "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61) 2) "After 22 sessions, a post-test was conducted to measure the degree of improvement in learners' reading comprehension at the end of the implementation period." (p. 61) Detailed Analysis: Criterion Y requires outcomes to be measured after at least 75% of a full academic year (roughly 9-10 months) from intervention start. The paper explicitly frames the intervention and follow-up as spanning "one semester" (22 sessions), which corresponds to roughly half, not three-quarters, of an academic year. No tracking beyond the semester-end post-test is reported. Criterion Y is not met because the tracking period was limited to one academic semester rather than at least 75% of an academic year.
  • B

    Balanced Control Group

    • Both groups received the same reading content, session count, and completion timeframe, differing only in the delivery medium (blog vs. paper), so resources were balanced.
    • "The main point that should be mentioned is that the control group received the same reading activities on the paper but the experimental group received them via blogs." (p. 61)
    • Relevant Quotes: 1) "During this period, the control group was given paper-based reading comprehension worksheet homework. Each worksheet corresponded to the topic and language of one unit of the course book." (p. 61) 2) "Both groups were given three days to complete the reading task homework." (p. 61) 3) "The reading activities were divided into several parts to be received via blogs everyday up to the next session." (p. 61) 4) "The main point that should be mentioned is that the control group received the same reading activities on the paper but the experimental group received them via blogs." (p. 61) 5) "Over one semester, both groups participated in the study for 22 sessions, and 10 reading comprehension texts were practiced in both groups." (p. 61) Detailed Analysis: Applying the Criterion B decision procedure: the variable being tested here is the delivery medium (blog vs. paper) for identical reading-comprehension content, not extra instructional time or budget, so EXTRA_RESOURCES_PRESENT is effectively false. Both groups practiced the same 10 reading comprehension texts across the same 22 sessions over one semester, and both were allotted the same three days to complete each homework task. The only documented difference is that the experimental group accessed the content through a blog (plus a one-time demonstration of blog use), while the control group received identical content on paper. No additional time, budget, or supplementary resource was given to the experimental group beyond the medium itself, and the control group received a matched, business-as-usual equivalent of the same activities on paper. Criterion B is met because both groups received identical reading content, session count, and completion timeframe, differing only in delivery medium.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study was found in the paper or via internet search of other research teams' work.
    • Relevant Quotes: 1) "Comparing the effectiveness of surfing blogs, the present study in line with several other studies (Brker & Torgesen, 1995; Mioduser, Tur-Kaspa, +& Leitner, 2000; Spezial & La-France, 1992; Attewell & Webster, 2004; Chinnery, 2006; Klopfer, Squire, & Jenkins, 2002; Soloway et al., 2001) found that surfing blogs is significantly effective in fostering EFL learners' reading comprehension." (p. 63) Detailed Analysis: The authors cite several prior studies as broadly consistent with technology/multimedia-assisted language learning in general, but none of these is presented as an independent replication of this specific 40-participant Iranian EFL study; most predate this 2017 paper and concern different populations, technologies (e.g., handheld devices, multimedia instruction), or skills. An internet search for later work by other author teams replicating this specific blog-exposure, 40-participant Kerman EFL reading-comprehension design (e.g., searches for citing/replication studies of this 2017 paper) did not surface any independent peer-reviewed replication; results instead returned unrelated blog/podcast-and-reading studies by different authors testing different designs and populations, which do not constitute a reproduction of this specific study. No subsequent peer-reviewed replication of this specific study (same intervention, same design, conducted by a different research team) was found. Criterion R is not met because no independent replication of this specific study by another research team was found.
  • A

    All-subject Exams

    • Because criterion E is not met and only reading comprehension (a single subject) was assessed, this criterion is not met.
    • "The reading comprehension test contained thirty five multiple-choice and open ended questions." (p. 60)
    • Relevant Quotes: 1) "The reading comprehension test contained thirty five multiple-choice and open ended questions." (p. 60) 2) "What effect does surfing blogs have on EFL learners' reading comprehension?" (p. 60) Detailed Analysis: Per the ERCT specification, criterion A cannot be met unless criterion E (standardised exam-based assessment) is first satisfied; since E was not met (the outcome measure was a custom, syllabus-linked test), A automatically fails. In addition, the study measured only reading comprehension and did not assess performance in any other core subject. Criterion A is not met because criterion E is not satisfied and only a single subject (reading) was assessed.
  • G

    Graduation Tracking

    • Criterion Y (a prerequisite) is not met, and no follow-up publication tracking this cohort toward graduation was found via internet search.
    • Relevant Quotes: 1) "After 22 sessions, a post-test was conducted to measure the degree of improvement in learners' reading comprehension at the end of the implementation period." (p. 61) Detailed Analysis: Per the ERCT specification, criterion G requires criterion Y to be met first; since Y was not met (tracking limited to one semester), G is automatically not met. In addition, there is no mention anywhere in the paper of tracking participants beyond the semester-end post-test, let alone through to graduation. An internet search for later publications by Najmeh Rafiee Moghadam and/or Mehry Haddad Narafshan that might track this same 40-participant Kerman cohort toward graduation did not find any such follow-up study; the authors' subsequent identifiable publications (e.g., work on critical thinking interventions and on education for sustainable development) concern different cohorts and different outcome constructs, not a graduation follow-up of this reading-comprehension sample. Criterion G is not met because criterion Y is not satisfied and no tracking beyond the immediate post-test, nor any follow-up publication, was found.
  • P

    Pre-Registered

    • The paper contains no statement of pre-registration, and no internet search evidence of a registered protocol for this study was found.
    • Relevant Quotes: No quotes referencing a trial registry, protocol registration, or pre-specified analysis plan were found anywhere in the paper. Detailed Analysis: There is no mention in the methods, procedure, or any other section of a registered protocol, a trial registry ID, or a pre-specified analysis plan published before data collection began. An internet search for a pre-registration record for this study (by title, authors, and topic) did not locate any registry entry; this is consistent with the paper's publication context (a small applied-linguistics RCT published in 2017, a period and field in which trial pre-registration was not standard practice). Criterion P is not met because no evidence of pre-registration is present in the paper or found via internet search.