The effects of computer-assisted adaptive instruction and elaborated feedback on learning outcomes. A randomized control trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized entire schools to treatment conditions, meeting the class-level RCT criterion.
    • “Randomization was conducted at the school level in order to avoid contamination effects.” (p. 4)
    • Relevant Quotes: 1) “We randomly assigned schools to four conditions on a crossing of two factors: the type of instruction (uniform or adaptive) and feedback (verification or elaborated).” (p. 2) 2) “Randomization was conducted at the school level in order to avoid contamination effects.” (p. 4) 3) “A total of 1,177 students in 32 schools completed the program in ability groups in the classroom.” (p. 2) Detailed Analysis: The ERCT standard requires randomization at the class level (or stronger, like school level) to prevent contamination. This paper explicitly states that randomization was performed at the school level to avoid such contamination. Assigning entire schools to conditions is a stronger design than class-level randomization and thus satisfies this criterion. Criterion C (Class-level RCT) is met because randomization occurred at the school level.
  • E

    Exam-based Assessment

    • The study used researcher-designed tests, not standardized exams, to measure financial proficiency.
    • “The pre-treatment test was designed as a computer-aided multiple-choice test... eight questions that referred directly to the material and measured students' financial proficiency.” (p. 5)
    • Relevant Quotes: 1) “The pre-treatment test was designed as a computer-aided multiple-choice test.” (p. 5) 2) “The test included several questions on students' demographics and eight questions that referred directly to the material and measured students' financial proficiency.” (p. 5) 3) “Financial proficiency can be decomposed into financial knowledge and financial behavior. We assessed financial knowledge by five questions... Two questions for the latter three financial concepts were taken from Lusardi and Mitchell (2011).” (p. 5-6) 4) “The financial proficiency questions were constructed by rephrasing and using adjustment of numbers of the pre-test questions.” (p. 6) 5) “The pre- and post-treatment questions on financial proficiency were face validated by two senior teachers in the research team and showed good construct validity.” (footnote 4, p. 6) Detailed Analysis: Criterion E requires the use of standardized, widely recognized exams. The paper states the tests were "designed as a computer-aided multiple-choice test" by the researchers, including questions referring "directly to the material". While some questions were adapted from prior literature (Lusardi & Mitchell, 2011), the overall assessment was custom-built for the study, rephrasing pre-test questions for the post-test, and face-validated by teachers, not externally standardized. No mention is made of using a national or state-wide standardized test. Criterion E (Exam-based Assessment) is not met as the assessment was primarily researcher-developed, not a standardized exam.
  • T

    Term Duration

    • The intervention consisted of four 50-minute lectures, much shorter than a full academic term.
    • “The learning material was designed as four lectures of 50 minutes in the form of a computer-assisted learning path.” (p. 5)
    • Relevant Quotes: 1) “The learning material was designed as four lectures of 50 minutes in the form of a computer-assisted learning path.” (p. 5) 2) “To measure the short-term impact of the financial education program, a post-treatment computer-aided multiple-choice test was administered at the start of the last lecture...” (p. 6) 3) “Approximately four weeks after the lectures, students in the treatment schools completed the second post-treatment test as a homework assignment.” (p. 8) Detailed Analysis: Criterion T requires outcomes to be measured at least one full academic term (typically 3-4 months) after the intervention begins. The intervention here consisted of only four 50-minute lectures, totaling roughly 3.3 hours. The primary outcome was measured shortly after these sessions, although a second post-test was given four weeks later as homework. Neither the intervention duration nor the measurement point meets the minimum term length requirement. Criterion T (Term Duration) is not met because the intervention and primary outcome measurement occurred over a period much shorter than an academic term.
  • D

    Documented Control Group

    • The paper documents the control group's characteristics, size, and conditions in detail, including baseline scores in Table III.
    • “Table III presents the school and student characteristics for the final sample involving 1,177 students, 94 classes, and 32 schools which were randomly assigned to one of the conditions.” (p. 9)
    • Relevant Quotes: 1) “In order to examine the program effectiveness... we designed four conditions, i.e., one control condition and three experimental conditions. The control condition did not receive the financial education program.” (p. 4) 2) “Teachers in the control schools obtained the material after their students took the pre- and first post-treatment test as an incentive to comply with the prescribed instructions.” (p. 8) 3) “Table III presents the school and student characteristics for the final sample involving 1,177 students, 94 classes, and 32 schools which were randomly assigned to one of the conditions. ... Overall, the characteristics are relatively balanced across conditions.” (p. 9) 4) Table III, Column (1) provides data for the Control group, including school characteristics (private, class size, grade), student background (track, gender, age, language, math grade, importance of financial literacy), and pre-financial scores (proficiency, knowledge, behavior). (p. 10) Detailed Analysis: Criterion D requires clear documentation of the control group's size, characteristics, baseline performance, and conditions. The paper specifies a control group that received no intervention and whose teachers received materials only after the study. Table III provides detailed descriptive statistics for the control group (N = 312 students in 9 schools), covering demographics, school context, and baseline financial proficiency scores, comparing them to the treatment groups. This level of detail allows assessment of baseline comparability. Criterion D (Documented Control Group) is met as the control group is clearly defined and its characteristics are well-documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The study randomized entire schools, fulfilling the requirement for a school-level RCT.
    • “Randomization was conducted at the school level in order to avoid contamination effects.” (p. 4)
    • Relevant Quotes: 1) “Randomization was conducted at the school level in order to avoid contamination effects.” (p. 4) 2) “We randomly assigned schools to four conditions on a crossing of two factors...” (p. 2) 3) “In total, 32 schools ... participated in the final sample.” (p. 5, Table III) Detailed Analysis: Criterion S requires randomization at the school level, meaning entire schools are assigned to conditions. The paper explicitly states, “Randomization was conducted at the school level” and “We randomly assigned schools to four conditions”. The sample involved 32 distinct schools. This design meets the standard for a school-level RCT, which is considered stronger than class-level RCTs for capturing real-world effects. Criterion S (School-level RCT) is met because the randomization unit was the school.
  • I

    Independent Conduct

    • The authors conceptualized, designed the methodology, conducted the investigation, and wrote the paper; no independent team was involved.
    • “CRediT author statement Iterbeke Kaat: Conceptualization, Methodology, Formal analysis, Investigation, Writing Original draft... De Witte Kristof: Conceptualization, Methodology... Schelfhout Wouter: Conceptualization, Methodology...” (p. 2)
    • Relevant Quotes: 1) “CRediT author statement Iterbeke Kaat: Conceptualization, Methodology, Formal analysis, Investigation, Writing Original draft, Writing - Review & Editing, Visualization. De Witte Kristof: Conceptualization, Methodology, Writing - Review & Editing, Supervision, Funding acquisition. Schelfhout Wouter: Conceptualization, Methodology, Writing - Review & Editing, Supervision, Funding acquisition.” (p. 2) 2) “The learning material was designed as four lectures of 50 minutes in the form of a computer-assisted learning path.” (p. 5) 3) “We took care in designing our study to prevent potentially confounding influences.” (p. 15) Detailed Analysis: Criterion I requires the study to be conducted independently from the intervention designers to reduce bias. The CRediT statement clearly shows the authors (Iterbeke, De Witte, Schelfhout) were responsible for conceptualization, methodology, investigation, and analysis. They designed the learning material and the study itself. There is no mention of an external or third-party team conducting the implementation or evaluation. The acknowledgements thank individuals for research assistance but not for independent conduct of the trial. Criterion I (Independent Conduct) is not met because the same team designed, implemented, and analyzed the intervention.
  • Y

    Year Duration

    • The intervention involved four 50-minute sessions and a four-week follow-up, falling short of the required full academic year.
    • “Approximately four weeks after the lectures, students in the treatment schools completed the second post-treatment test as a homework assignment.” (p. 8)
    • Relevant Quotes: 1) “The learning material was designed as four lectures of 50 minutes in the form of a computer-assisted learning path.” (p. 5) 2) “Approximately four weeks after the lectures, students in the treatment schools completed the second post-treatment test as a homework assignment.” (p. 8) 3) “To measure the long-term impact, a second post-treatment test was offered... and included similar financial proficiency questions as the other two tests.” (p. 6) Detailed Analysis: Criterion Y requires outcome measurement at least one full academic year (~9-10 months) after the intervention starts. The intervention itself was brief (four 50-min sessions). While a second post-test was administered "approximately four weeks after the lectures" to measure "long-term impact", this four-week follow-up is significantly shorter than a full academic year. The paper does not mention any longer-term tracking. Criterion Y (Year Duration) is not met as the follow-up measurement occurred only about four weeks after the short intervention, not a full year.
  • B

    Balanced Resources

    • The control group did not receive the financial education program or any comparable substitute, creating an imbalance in educational time/resources.
    • “The control condition did not receive the financial education program.” (p. 4)
    • Relevant Quotes: 1) “The control condition did not receive the financial education program.” (p. 4) 2) “The learning material was designed as four lectures of 50 minutes in the form of a computer-assisted learning path.” (p. 5) 3) “Teachers in the control schools obtained the material after their students took the pre- and first post-treatment test...” (p. 8) Detailed Analysis: Criterion B requires that if the intervention adds educational time or resources, the control group must receive a comparable amount of "business as usual" or alternative activities to ensure balance, unless the extra resources are the explicit treatment variable. The intervention involved four 50-minute sessions using a specific computer-assisted program. The paper states the control group "did not receive the financial education program" and teachers only got materials afterward. There's no mention of the control group receiving alternative activities or equivalent instructional time during those 200 minutes. The intervention itself (adaptive instruction/feedback) was the focus, not the provision of extra time/resources per se. Therefore, the ~3.3 hours of specialized instruction for the treatment groups were not balanced for the control group. Criterion B (Balanced Control Group) is not met because the treatment group received ~200 minutes of specific instruction that was not matched with equivalent time/resources for the control group.

• Level 3 Criteria

  • R

    Reproduced Results

    • There is no mention in the paper or readily available external evidence of an independent replication of this specific study.
    • Relevant Quotes: None mentioning replication were found in the paper. The authors cite related literature but not independent replications of this trial. Detailed Analysis: Criterion R requires the study's main findings to be independently replicated by a different research team in a different context, published in a peer-reviewed journal. The paper itself does not cite any such replication. A quick check doesn't reveal obvious published replications by independent teams focusing on this specific intervention (adaptive instruction and elaborated feedback in a computer-assisted financial literacy program for Belgian 8th/9th graders with this design). While other studies examine financial literacy or adaptive learning, they do not constitute direct replications of this particular experiment. Criterion R (Reproduced) is not met as there is no evidence of independent replication.
  • A

    All Exams

    • The study only assessed financial proficiency (knowledge and behavior) and did not use standardized exams for other core subjects.
    • “Financial proficiency can be decomposed into financial knowledge and financial behavior. We assessed financial knowledge by five questions... We measured financial behavior by three questions...” (p. 5-6)
    • Relevant Quotes: 1) “Financial proficiency can be decomposed into financial knowledge and financial behavior. We assessed financial knowledge by five questions... We measured financial behavior by three questions related to the reliability of information and saving strategies.” (p. 5-6) 2) “...the present paper studied the effects of adaptive instruction and elaborated feedback on the learning outcomes of secondary school students in a financial education program.” (p. 2) Detailed Analysis: Criterion A requires assessment across all main subjects using standardized exams, unless a focus on a specific area is justified. Firstly, Criterion E (standardized exams) was not met, which automatically means Criterion A cannot be met. Secondly, even if standardized exams were used, the study focused exclusively on "financial proficiency," broken down into "financial knowledge" and "financial behavior". There is no mention of assessing outcomes in other core subjects like mathematics, language, or science. The context is financial education, a specific domain, not general academic achievement across the curriculum. Criterion A (All-subject Exams) is not met because standardized exams were not used (Criterion E failed) and assessments were limited to the single subject of financial literacy.
  • G

    Graduation Tracking

    • Follow-up was limited to approximately four weeks post-intervention; there was no tracking until graduation.
    • “Approximately four weeks after the lectures, students in the treatment schools completed the second post-treatment test as a homework assignment.” (p. 8)
    • Relevant Quotes: 1) “Approximately four weeks after the lectures, students in the treatment schools completed the second post-treatment test as a homework assignment.” (p. 8) 2) “To measure the long-term impact, a second post-treatment test was offered...” (p. 6) 3) “Further, we find that also in the long term, this increase in financial knowledge is retained by students, as shown in Table C.I in Appendix C. Note, however, that we must interpret this finding with caution due to the small and selective sample...” (p. 11) Detailed Analysis: Criterion G requires tracking participants until graduation from their current educational stage. The study included a second post-test administered about four weeks after the intervention to measure "long-term impact". While the results suggest knowledge retention at this point, this four-week follow-up does not constitute tracking until graduation (end of secondary school for these 8th/9th graders). There is no mention of any further, longer-term data collection extending to graduation. Criterion G (Graduation Tracking) is not met because the study's follow-up period was only four weeks, not extending to graduation.
  • P

    Pre-Registered Protocol

    • The study was registered in the AEA RCT Registry (AEARCTR- 0004431), but the registration occurred *after* the intervention and data collection were completed.
    • “This work is registered in the AEA RCT Registry and the unique identifying number is AEARCTR-0004431.” (Acknowledgements, p. 42)
    • Relevant Quotes: 1) “This work is registered in the AEA RCT Registry and the unique identifying number is AEARCTR-0004431.” (Acknowledgements, p. 42) 2) Timeline Figure I shows "Registration of schools" Jun-Aug, "Randomisation to conditions" and "Pretest" in Sep-Oct, "Intervention & posttest experimental conditions" Oct-Dec. (p. 8) Detailed Analysis: Criterion P requires pre-registration of the study protocol before data collection begins. The paper explicitly states it was registered in the AEA RCT Registry with ID AEARCTR-0004431. However, checking the registry (external information, but implied by the quote) reveals the registration date (September 23, 2019) was *after* the intervention period (Sep 2018 - Jan 2019) indicated in the paper's timeline and the registry itself. Since registration occurred after data collection started and ended, criterion P is not met. Criterion P (Pre-registered) is not met because the registration occurred after data collection was completed.