A randomized controlled trial of project-based learning for middle-school financial literacy

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized at the intact class-section level (two classes), satisfying the class-level RCT requirement.
    • "Two intact class sections were randomly assigned to the Standards-Based Learning (SBL) group (n = 31; 8 female; ages 11–13, M = 11.94) or the Project-Based Learning (PBL) group (n = 32; 12 female; ages 11–13, M = 11.93)." (p. 3)
    • Relevant Quotes: 1) "This class-randomized, posttest-only trial (two intact sections) compared Project-Based Learning and Standards-Based Learning in a budgeting unit delivered via Nearpod® across three 40-min class sessions." (p. 1) 2) "Two intact class sections were randomly assigned to the Standards-Based Learning (SBL) group (n = 31; 8 female; ages 11–13, M = 11.94) or the Project-Based Learning (PBL) group (n = 32; 12 female; ages 11–13, M = 11.93)." (p. 3) 3) "Classes were randomly assigned to one of two conditions: the 7th-period class was assigned to the Project-Based Learning condition, and the 3rd-period class was assigned to the Standards-Based Learning condition." (p. 3) Detailed Analysis: Criterion C requires that the unit of randomization be the class (or a stronger unit such as the school), to reduce contamination that can occur when students within the same class receive different conditions. The paper explicitly describes a "class-randomized" trial with "two intact sections" and states that the classes (periods) were assigned to different conditions. This indicates the allocation occurred at the class-section level rather than within a single class at the student level. Criterion C is met because intact class sections were assigned to conditions, which satisfies class-level randomization.
  • E

    Exam-based Assessment

    • Outcomes were assessed with a researcher-created posttest and rubric rather than a standardized exam-based assessment.
    • "The full measure was created by the research team and reviewed by instructional staff to establish face and content validity." (p. 3)
    • Relevant Quotes: 1) "Students completed a 12-item posttest in Microsoft Forms that included seven multiple-choice items, three short-answer definitions, and two scenario-based open responses." (p. 3) 2) "Open-ended responses (five items) were scored using a rubric developed by the research team assessing accuracy, correct use of budgeting vocabulary, and logical financial reasoning." (p. 3) 3) "A small pilot with five middle school students was conducted to refine wording and instructions prior to implementation." (p. 3) 4) "The full measure was created by the research team and reviewed by instructional staff to establish face and content validity." (p. 3) Detailed Analysis: Criterion E requires that outcomes be measured using a standardized, widely recognized exam-based assessment (for example, state/national standardized tests or established standardized instruments). The paper describes a 12-item posttest administered in Microsoft Forms and explicitly states that the measure was "created by the research team," with scoring based on a rubric "developed by the research team." These statements indicate a custom, study-specific assessment rather than a standardized exam. Criterion E is not met because the outcome measure was created for this study rather than being a standardized exam-based assessment.
  • T

    Term Duration

    • Outcomes were measured immediately after a three-session unit, which is far shorter than a full academic term after start.
    • "The intervention took place over three 40-min class periods during regularly scheduled instructional time at the same public middle school." (p. 3)
    • Relevant Quotes: 1) "This class-randomized, posttest-only trial (two intact sections) compared Project-Based Learning and Standards-Based Learning in a budgeting unit delivered via Nearpod® across three 40-min class sessions." (p. 1) 2) "The intervention took place over three 40-min class periods during regularly scheduled instructional time at the same public middle school." (p. 3) 3) "Students completed a 12-item posttest in Microsoft Forms ..." (p. 3) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (typically about 3–4 months) after the intervention begins. The study is described as "posttest-only" and as occurring across "three 40-min class sessions," which implies assessment occurred immediately after the short intervention rather than after term-long follow-up. The paper does not report any later follow-up measurement at a term-long interval. Criterion T is not met because the study spans only three class sessions with posttest-only measurement, not term-long follow-up.
  • D

    Documented Control Group

    • The control condition is described, but baseline academic performance is not documented because the design is posttest-only and prior knowledge was not measured.
    • "All students received the same test across the Project-Based Learning and Standards-Based Learning groups, and no prior budgeting knowledge was assumed or measured." (p. 3)
    • Relevant Quotes: 1) "This class-randomized, posttest-only trial (two intact sections) ..." (p. 1) 2) "Two intact class sections were randomly assigned to the Standards-Based Learning (SBL) group (n = 31; 8 female; ages 11–13, M = 11.94) or the Project-Based Learning (PBL) group (n = 32; 12 female; ages 11–13, M = 11.93)." (p. 3) 3) "In the Standards-Based Learning condition, students were taught through recorded PowerPoint-style lectures that presented budgeting concepts in a structured, sequential manner." (p. 3) 4) "All students received the same test across the Project-Based Learning and Standards-Based Learning groups, and no prior budgeting knowledge was assumed or measured." (p. 3) Detailed Analysis: Criterion D requires that the control group be well documented, including (at minimum) clear description of the control condition and documentation that supports comparability (including baseline performance information). The paper documents the control group size and demographics (age range, mean age, and gender counts) and provides a substantive description of the Standards-Based Learning control condition. However, the study is explicitly "posttest-only," and it also states that "no prior budgeting knowledge was assumed or measured." Therefore, the report does not provide baseline academic performance (either budgeting or general achievement) for the control group, which limits the documentation needed for assessing baseline equivalence. Criterion D is not met because baseline performance information for the control group is not documented in this posttest-only design.

• Level 2 Criteria

  • S

    School-level RCT

    • The trial was conducted within a single school and randomized between two classes, not between schools.
    • "Sixty-three sixth-grade students from the same public middle school in the United States participated." (p. 3)
    • Relevant Quotes: 1) "Sixty-three sixth-grade students from the same public middle school in the United States participated." (p. 3) 2) "Two science classes at the school participated: the 3rd-period and the 7th-period classes." (p. 3) 3) "Classes were randomly assigned to one of two conditions ..." (p. 3) Detailed Analysis: Criterion S requires school-level randomization, meaning schools (or equivalent sites) are assigned to intervention and control. The paper reports one public middle school and two class periods within that school, with assignment occurring at the class level. There is no evidence of multiple schools or school-level assignment. Criterion S is not met because randomization occurred between classes within a single school rather than between schools.
  • I

    Independent Conduct

    • The paper indicates the study design, data collection, and analysis were conducted by the author(s) without an independent external evaluation team.
    • "All aspects of study design, data collection, statistical testing, interpretation, and manuscript writing were conducted independently by me." (p. 8)
    • Relevant Quotes: 1) "All aspects of study design, data collection, statistical testing, interpretation, and manuscript writing were conducted independently by me." (p. 8) 2) "IM: Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Resources, Software, Visualization. GO: Supervision, Writing – review & editing, Data curation, Formal Analysis, Visualization, Writing – original draft." (p. 7) 3) "Open-ended responses (five items) were scored using a rubric developed by the research team ..." (p. 3) Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the intervention designers/authors, to reduce bias in implementation, measurement, analysis, and reporting. The Generative AI statement says that the author conducted "study design, data collection, statistical testing, interpretation, and manuscript writing" themselves. The paper also notes the scoring rubric was developed by the research team. While the use of standardized, pre-scripted Nearpod materials may reduce some implementation variability, the paper does not document an independent third-party evaluation team for data collection or analysis. Criterion I is not met because the study does not document independent conduct by an external evaluator.
  • Y

    Year Duration

    • The study spans only three class sessions and therefore does not track outcomes for at least 75% of an academic year; also, T is not met so Y cannot be met.
    • "The intervention took place over three 40-min class periods during regularly scheduled instructional time at the same public middle school." (p. 3)
    • Relevant Quotes: 1) "This class-randomized, posttest-only trial (two intact sections) compared Project-Based Learning and Standards-Based Learning in a budgeting unit delivered via Nearpod® across three 40-min class sessions." (p. 1) 2) "The intervention took place over three 40-min class periods during regularly scheduled instructional time at the same public middle school." (p. 3) Detailed Analysis: Criterion Y requires outcome measurement at least 75% of an academic year after the intervention begins. This study is a short unit delivered across three 40-minute class periods with a posttest-only design, which is far shorter than an academic year. Additionally, per the ERCT rule provided for this task, if criterion T is not met then criterion Y is not met. Since T is not met here, Y cannot be met. Criterion Y is not met because the study duration is far shorter than an academic year and it fails the term-duration prerequisite.
  • B

    Balanced Control Group

    • Both groups used the same scheduled class time and the same Nearpod delivery platform, with no evidence of additional time or budget provided only to the intervention group.
    • "Instruction in both conditions was delivered through Nearpod, a standardized digital platform using pre-scripted, automated materials with minimal teacher intervention." (p. 3)
    • Relevant Quotes: 1) "The intervention took place over three 40-min class periods during regularly scheduled instructional time at the same public middle school." (p. 3) 2) "Instruction in both conditions was delivered through Nearpod, a standardized digital platform using pre-scripted, automated materials with minimal teacher intervention." (p. 3) 3) "In the Project-Based Learning condition, students completed a simulation to design a class trip under a fixed USD 1,000 budget (see Figure 2 and Table 1)." (p. 3) 4) "In the Standards-Based Learning condition, students were taught through recorded PowerPoint-style lectures that presented budgeting concepts in a structured, sequential manner." (p. 3) Detailed Analysis: Criterion B compares the nature, quantity, and quality of resources (time, materials, staffing/support) provided to intervention and control conditions, and asks whether the control condition provides a comparable substitute for the intervention’s inputs unless extra resources are explicitly the treatment. The paper states the intervention occurred during "regularly scheduled instructional time" across the same "three 40-min class periods" and that "both conditions" used Nearpod with pre-scripted automated materials. This supports that instructional time and the main delivery platform were equivalent across conditions. The main difference is pedagogical approach (simulation/project task versus lecture/practice). The mention of a "$ 1,000 budget" in the PBL simulation is part of the instructional scenario, not an indication of additional real funding or instructional time. Criterion B is met because the paper documents comparable class time and delivery resources across groups, with no evidence of unbalanced additional time or budget given only to the PBL group.

• Level 3 Criteria

  • R

    Reproduced

    • No independent, peer-reviewed replication of this specific trial by other authors was found in the paper or via internet search.
    • Relevant Quotes: 1) "To date, no known study has conducted a randomized controlled trial using Project-Based Learning as the primary instructional intervention for teaching financial literacy in a middle school setting." (p. 2) Detailed Analysis: Criterion R requires evidence that an independent research team replicated this study (or a clearly equivalent trial of the same intervention comparison) in another context and published the replication in a peer-reviewed outlet. The target paper does not report that its findings have been replicated; instead, it emphasizes the novelty of RCT evidence in this specific area. I also searched the internet using the paper title and DOI for replication studies by other authors. I did not find any peer-reviewed publications that explicitly claim to replicate this specific two-class Nearpod-delivered budgeting unit trial. Criterion R is not met because independent replication evidence was not found.
  • A

    All-subject Exams

    • Because E is not met and outcomes are measured only with a budgeting posttest (not standardized exams across core subjects), A is not met.
    • "Students completed a 12-item posttest in Microsoft Forms that included seven multiple-choice items, three short-answer definitions, and two scenario-based open responses." (p. 3)
    • Relevant Quotes: 1) "Students completed a 12-item posttest in Microsoft Forms that included seven multiple-choice items, three short-answer definitions, and two scenario-based open responses." (p. 3) 2) "Items targeted (1) conceptual knowledge, (2) categorical reasoning, and (3) applied budgeting decisions, all aligned to lesson objectives." (p. 3) Detailed Analysis: Criterion A requires standardized exam-based assessments across all main subjects, and per the ERCT rule provided for this task, if criterion E is not met then criterion A is not met. This study uses a researcher-created budgeting posttest aligned to lesson objectives rather than any standardized exam, and it does not assess outcomes across core subjects (e.g., math, reading, science) using standardized tests. Criterion A is not met because the study fails E and does not use all-subject standardized exams.
  • G

    Graduation Tracking

    • The paper reports no long-term follow-up and, because Y is not met, G cannot be met; no follow-up graduation-tracking papers by the same authors were found via internet search.
    • "Furthermore, we did not measure long-term outcomes, leaving it unclear whether gains in budgeting comprehension persist beyond the immediate posttest." (p. 6)
    • Relevant Quotes: 1) "Furthermore, we did not measure long-term outcomes, leaving it unclear whether gains in budgeting comprehension persist beyond the immediate posttest." (p. 6) Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. The paper explicitly states it did not measure long-term outcomes, which indicates outcomes were not tracked beyond the immediate posttest. Additionally, per the ERCT rule provided for this task, if criterion Y is not met then criterion G is not met. Since Y is not met for this study, G cannot be met. I also searched for subsequent papers by the same authors (Mishra and Orosz) reporting longer-term follow-up of the same cohort through graduation, but I did not find any such publications as of the ERCT check date. Criterion G is not met because the study includes no long-term graduation tracking and Y is not met.
  • P

    Pre-Registered

    • The paper provides no pre-registration link, registry ID, or registration date, and no registry record was found via search.
    • Relevant Quotes: (No text in the paper mentions pre-registration, a registry platform, a registration ID, or a registration date.) Detailed Analysis: Criterion P requires that the study protocol be pre-registered before data collection begins, typically including a registry platform and an identifier/date. Searching the paper PDF for common registration terms (e.g., "pre-registration," "registration," "OSF") did not find any pre-registration statement. I also searched the internet using the paper title and DOI for any linked registration record (e.g., OSF Registries, AEA RCT Registry), but did not find a matching pre-registered protocol. Criterion P is not met because there is no reported or located evidence of pre-registration.