Effect of the Paideia Seminar on the Comprehension of Poetry and Reading Anxiety

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Two entire intact classes, not individual students within one class, were randomly assigned to the control and experimental conditions.
    • "Two intact classes (n = 50) of grade 9 learners of English as a foreign language were randomly assigned to control and experimental conditions." (p. 74)
    • Relevant Quotes: 1) "The study employed a pre-test - post-test control group design whereby two intact classes were randomly assigned to control and experimental conditions." (p. 69) 2) "Two intact classes (n = 50) of grade 9 learners of English as a foreign language were randomly assigned to control and experimental conditions." (p. 74) 3) "A sample of 50 EFL learners enrolled in two sections of grade 9 was randomly assigned to control and experimental conditions." (p. 75) Detailed Analysis: The unit of randomisation here is the intact class/section, not the individual student. The authors explicitly state that two whole sections (classes) of grade 9 were assigned as units to the control and experimental conditions, rather than splitting students within a single classroom into two groups. This design avoids the specific contamination problem the criterion is meant to guard against (students in the same room receiving different conditions), because each class as a whole received only one condition. This is not a school-level RCT (only one school and two sections were involved, so the stronger S criterion is not met), but it clearly satisfies the weaker class-level requirement. Criterion C is met because randomisation occurred at the level of two whole intact classes rather than within a single class.
  • E

    Exam-based Assessment

    • The reading comprehension tests were adapted from a Paideia lesson-plan website rather than a standardised, validated exam, and no standardised exam was used for the anxiety outcome either.
    • "Specifically, we used part one of the test given by Paideia Active Learning" ... on Shakespeare's 'The Seven Ages of Man' as a pre-test and a part of the "Mending Wall" reading questions retrieved from Paideia Active Learning as well ... as the post tests." (p. 74)
    • Relevant Quotes: 1) "Two reading achievement tests were used as pre-test and post-test measures of reading comprehension. Specifically, we used part one of the test given by Paideia Active Learning" (https://www.paideia.org/lesson-plans/seven-ages- of-man/) on Shakespeare's 'The Seven Ages of Man' as a pre-test and a part of the "Mending Wall" reading questions retrieved from Paideia Active Learning as well (https://www.paideia.org/lesson- plans/mending-wall/) as the post tests (See Appendix C)." (p. 74) 2) "Both tests consisted of response and comprehension question items and assessed the main idea, literal, analysis, synthesis, inference, application, and creative comprehension." (p. 74) 3) "In addition, the Reading Anxiety Scale (Young, 1999) was used to assess the readers' levels of reading anxiety prior to and after the treatment. This scale consists of 15 Likert-type items, 7 of which are negatively stated to avoid a response set." (p. 74) 4) Appendix C reproduces both tests, which consist of open-ended discussion-style comprehension questions (e.g., "What images comes to mind when you read about the different 'ages of man'...?") taken directly from downloadable lesson-plan question sets on the paideia.org website, not from any national, state, or otherwise psychometrically validated standardised examination. (p. 85-89) Detailed Analysis: The ERCT E criterion requires a widely recognised, standardised exam-based assessment rather than a study-specific or informally sourced instrument. Here the reading achievement pre-test and post-test are lesson-plan comprehension question sets downloaded from the Paideia Active Learning website, tied to two specific poems ("The Seven Ages of Man" and "Mending Wall"). These are teacher resource materials designed to accompany lesson plans, not a standardised, psychometrically validated exam with established norms, reliability, or validity evidence. The paper reports no information on the instrument's standardisation, norming population, or psychometric properties beyond noting the general comprehension categories assessed. The Reading Anxiety Scale (Young, 1999) is a published affective scale, but it measures anxiety, not an academic exam outcome, and the paper itself reports only "moderately high" internal consistency (alpha = .611) for this specific sample, which is not evidence of a widely validated standardised instrument's use here either. Because the core reading comprehension outcome was measured with a non-standardised, lesson-plan- derived instrument, Criterion E is not satisfied.
  • T

    Term Duration

    • The intervention and outcome measurement together spanned only six weeks, far short of a full academic term.
    • "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76)
    • Relevant Quotes: 1) "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76) 2) "Each seminar was conducted during 60 minutes for the seminar itself and 20 minutes for group evaluation at the end." (p. 77) 3) "The ANCOVA results of the analysis of the experimental versus control group post-test reading achievement scores, after having controlled for pre-test scores existing differences, were found to be statistically significant..." (p. 77) — the post-test was administered at the end of the same 6-week treatment period described above, with no later follow-up measurement reported. Detailed Analysis: The entire intervention, from pre-test through treatment to post-test, took place over six weeks at five hours per week. There is no indication that outcomes were measured any later than immediately following the conclusion of the six-week treatment period. A full academic term is typically defined as roughly 3-4 months; six weeks is well under half of that minimum, and no evidence in the paper suggests a longer interval between intervention start and outcome measurement. Because the study duration falls well short of the one-term minimum required, and because the study is also far short of a full year (so the stronger Y criterion cannot be met either), Criterion T is not met.
  • D

    Documented Control Group

    • The control group's regular-instruction procedures, activities, and shared demographic/baseline characteristics with the experimental group are described in reasonable detail.
    • "Instruction in the control group consisted of regular reading comprehension practice based on the pedagogical implications of the interactive theory of reading which requires focus on the different steps of the reading comprehension process." (p. 76)
    • Relevant Quotes: 1) "Meanwhile, participants in the control group read the same texts according the procedures of regular reading comprehension instruction." (p. 76) 2) "Instruction in the control group consisted of regular reading comprehension practice based on the pedagogical implications of the interactive theory of reading which requires focus on the different steps of the reading comprehension process. Specifically, instruction in the control group consisted or pre-reading, during reading, post reading stages in which a range of activities were used in order to activate readers' background knowledge, build vocabulary, check comprehension, and reflect on what is read. Examples of the activities used in the control group include brainstorming based on titles and illustrations; vocabulary learning strategies such as structural analysis, guessing meaning from context, and dictionary; literal and higher-order comprehension checks; and reflection." (p. 76) 3) "As such, the experimental and the control group included EFL learners from economically underprivileged families and were all native speakers of Arabic. Thirty (n = 37) participants were Lebanese with limited English language proficiency and 13 (n = 13) participants are Syrian refugees with comparable limited-English proficiency ... The age of the participants ranged from 14-16 years." (p. 75) 4) "Participants' pre-test scores on the dependent variables under investigation were as covariates in order to mathematically adjust for potential pre-existing difference between the control and the experimental group." (p. 74) Detailed Analysis: The paper describes, in specific procedural detail, what the control group actually experienced (pre-reading/during-reading/post-reading activities such as brainstorming, vocabulary strategies, and comprehension checks), rather than a vague "business as usual" label. It also documents the shared demographic profile (nationality mix, socio-economic background, native language, age range) of the overall sample from which both intact classes were drawn, and it uses pre-test scores as covariates specifically to characterise and adjust for baseline differences between the control and experimental groups. While the paper does not give a separate numeric breakdown of the control group's size or demographics apart from the experimental group, the combination of a detailed activity description and baseline-score covariate adjustment provides adequate documentation to assess comparability. Criterion D is met because the control condition's instructional activities and the sample's baseline characteristics are clearly and specifically documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred between two classes within a single school, not between multiple schools.
    • "This study was conducted at one of the typical public schools in Beirut city." (p. 75)
    • Relevant Quotes: 1) "This study was conducted at one of the typical public schools in Beirut city. The student population is approximately 800 students, 72% of whom are Lebanese and 28% are Syrian." (p. 75) 2) "A sample of 50 EFL learners enrolled in two sections of grade 9 was randomly assigned to control and experimental conditions." (p. 75) Detailed Analysis: The entire study took place at a single public school in Beirut, using two intact sections (classes) of grade 9 within that one school as the units of random assignment. There is no mention of any other school, and no cross-school comparison or randomisation. This is a single-school, two-class design, which is precisely the class-level (not school-level) design the standard distinguishes. Criterion S is not met because randomisation was confined to two classes within one school rather than across multiple schools.
  • I

    Independent Conduct

    • The authors themselves acknowledge that one of the researchers personally taught both the control and the experimental group, so the study was not conducted by an independent, third-party team.
    • "One of the researchers taught both the control and the experimental group, which may have affected the study results." (p. 80)
    • Relevant Quotes: 1) "Finally, further research is recommended in order to determine the generalizability of these findings ... This is especially so given that the present study employed a convenient sample of participants and one of the researchers taught both the control and the experimental group, which may have affected the study results." (p. 80) 2) "The teacher of the experimental group acted as the facilitator of the Paideia Seminar." (p. 76) — no separate, external, or blinded evaluator is described anywhere in the Methods section; the same research team designed, delivered, and analysed the study. Detailed Analysis: The Independent Conduct criterion requires that the study be conducted by a team separate from the intervention designers, to reduce the risk of biased implementation or reporting. Here, the authors explicitly disclose, as a limitation, that one of the two researchers personally delivered instruction to both the treatment and the control class. There is no mention anywhere of an external evaluator, independent data collector, or third-party administering agency; the same small research team appears to have designed the intervention, taught it, and analysed the resulting data. Because the authors themselves flag this direct, acknowledged lack of independence as a limitation affecting the results, Criterion I is clearly not met.
  • Y

    Year Duration

    • Because the Term Duration criterion (T) is not met, the stronger Year Duration criterion is automatically not met as well.
    • "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76)
    • Relevant Quotes: 1) "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76) Detailed Analysis: Per the ERCT decision rules, if the weaker Term Duration criterion (T) is not met, the stronger Year Duration criterion (Y) cannot be met either. The six-week treatment and measurement window here is dramatically shorter than the roughly 75%-of-a-year threshold (about 7 months) required for Y, so this criterion fails independently of the gating rule as well. Criterion Y is not met both because Criterion T is not met and because six weeks is far short of the year-long tracking requirement in its own right.
  • B

    Balanced Control Group

    • Both groups received the same amount of instructional time on the same texts and curriculum; only the teaching method (Paideia Seminar dialogue versus regular comprehension instruction) differed, so no extra time or budget was given to the experimental group that needed to be matched.
    • "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76)
    • Relevant Quotes: 1) "The experimental group participants read the assigned texts during the treatment period using the procedures of the Paideia Seminar ... Meanwhile, participants in the control group read the same texts according the procedures of regular reading comprehension instruction." (p. 76) 2) "All the participants received the treatment for a period of 6 weeks at the rate of 5 hours per week." (p. 76) 3) "Instruction in the control group consisted of regular reading comprehension practice ... a range of activities were used in order to activate readers' background knowledge, build vocabulary, check comprehension, and reflect on what is read." (p. 76) Detailed Analysis: Applying the (updated) Criterion B decision tree, the first check is whether the intervention group received extra time or budget compared to the control group (EXTRA_RESOURCES_PRESENT). Here, both the experimental and control classes read the same assigned texts over the identical schedule of 6 weeks at 5 hours per week; the only difference is the instructional method used to teach those texts (dialogic Paideia Seminar with pictures versus a standard pre-reading/during-reading/post-reading comprehension approach). No additional class time, materials budget, or resources were allocated to the experimental group beyond the method itself. Since EXTRA_RESOURCES_PRESENT is false (no additional time or budget was given to either group; the comparison is a method-versus-method contrast on equal footing), the decision tree resolves at its first branch: "if (!EXTRA_RESOURCES_PRESENT) return met()." There is no need to reach the integral-vs-non-integral or within-subjects branches, since no extra resource exists to classify. Criterion B is met because the two conditions were allocated identical instructional time and worked from the same texts, differing only in teaching method, not in resource quantity.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study by a different research team, published in a peer-reviewed journal, was reported in the paper or found through internet searches.
    • Relevant Quotes: No quotes describing an independent replication of this specific study (same intervention, same or a comparable population, conducted by a different research team) appear anywhere in the paper. The Discussion section compares the results to prior literature on Paideia/Socratic Seminars generally (e.g., Roberts, 2009; Adler, 1982) but these are cited as prior theoretical/empirical support for the Seminar approach in general, not as replications of this Lebanese EFL poetry-comprehension study's design and findings. Detailed Analysis: Criterion R requires that this specific study (or its central experimental claim, in a comparable design) be independently replicated by a different research team and published in a peer-reviewed outlet. Internet searches did not surface a peer-reviewed replication of this specific design (RCT of Paideia Seminar vs. regular instruction on EFL ninth-grade poetry comprehension and reading anxiety in Lebanon). One tangentially related item was located — a graduate dissertation titled "A Study to Examine the Impact of the Paideia Seminar Reading Intervention Program at a School in Connecticut" — which appears to discuss the Paideia Seminar's effect on reading comprehension in a different (U.S., non-EFL) school context; however, the source page could not be accessed (HTTP 403) to extract verbatim quotes or confirm whether it cites or replicates this specific study's design, so no quote from it is included here per instructions not to fabricate quotes. Even if accessible, a graduate dissertation is not confirmed to be a peer-reviewed publication, and its research question ("How can students' engagement in efferent and aesthetic reading through the Paideia Seminar impact students' reading comprehension skills?") suggests a different, broader design rather than a direct replication of this study's specific RCT, poetry texts, and reading anxiety outcome. Criterion R is not met because no independently verifiable, peer-reviewed replication of this specific study was found.
  • A

    All-subject Exams

    • Because Criterion E (standardised exam-based assessment) is not met, and only reading/language arts outcomes were measured (not all core subjects), Criterion A is not met.
    • "This article reports the results of an experimental study on the relative effectiveness of the Paideia Seminar in improving the comprehension of poetry and decreasing reading anxiety." (p. 69)
    • Relevant Quotes: 1) "This article reports the results of an experimental study on the relative effectiveness of the Paideia Seminar in improving the comprehension of poetry and decreasing reading anxiety." (p. 69) 2) "Two reading achievement tests were used as pre-test and post-test measures of reading comprehension." (p. 74) Detailed Analysis: Per the ERCT rule for Criterion A, if the exam-based assessment criterion (E) is not met, then A is automatically not met. In addition, the study only assessed reading/poetry comprehension and reading anxiety; no other core subjects (mathematics, science, etc.) were measured, and no rationale for restricting assessment to this single subject as an acceptable specialised-track exception is offered. Criterion A is not met both because Criterion E is not met and because only one subject area (reading) was assessed.
  • G

    Graduation Tracking

    • Because Criterion Y (Year Duration) is not met, and no follow-up beyond the six-week study period is reported or found, Criterion G is not met.
    • Relevant Quotes: No quotes describing any follow-up data collection after the six-week treatment and immediate post-test appear in the paper. The Conclusion recommends "further research ... to determine the generalizability of these findings" (p. 80) but does not report or reference any planned or completed longer-term or graduation-tracking follow-up of this cohort. Detailed Analysis: Per the ERCT rule for Criterion G, if Criterion Y is not met, then G is automatically not met. Consistent with this, the paper reports outcome measurement only immediately following the six-week intervention, with no indication of tracking participants through to graduation. Internet searches for subsequent publications by Awada and/or Ghaith tracking this same ninth-grade cohort through to graduation did not surface any such follow-up study. Criterion G is not met because Criterion Y is not met and no graduation-tracking follow-up publication was found.
  • P

    Pre-Registered

    • The paper contains no statement of pre-registration on any registry platform, and no pre-registration record for this study was found online.
    • Relevant Quotes: No quotes referencing a pre-registration registry, protocol publication, or registration date appear anywhere in the paper, including the Methods section and References list. Detailed Analysis: Criterion P requires evidence that the study's full protocol, including hypotheses, methods, and planned analyses, was publicly registered before data collection began. The paper makes no mention of any pre-registration platform, registry ID, or registration date at any point, and an internet search for a pre-registration record tied to this study (by title, authors, or DOI) did not return any matching registry entry. Criterion P is not met because no evidence of pre-registration is present in the paper or found through external search.