Online Mathematics Homework Increases Student Achievement

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the class (school) level, avoiding within-class mixing and meeting the class-level RCT standard.
    • “We investigated this question through a randomized controlled experiment in which schools were assigned to either an intervention or a business-as-usual condition.” (p. 3)
    • Relevant Quotes: 1) “We investigated this question through a randomized controlled experiment in which schools were assigned to either an intervention or a business-as-usual condition.” (p. 3) 2) “All schools that successfully completed an application were accepted into the study and randomly assigned to a condition, as described later.” (p. 4) Detailed Analysis: The paper clearly states that entire schools (and thus all their classes) were randomly assigned to treatment or control. This prevents any contamination within classes and satisfies the ERCT “Class-level RCT” requirement. Final sentence: Criterion C is met because randomization was done at the class/school level rather than at the student level.
  • E

    Exam-based Assessment

    • Student learning was measured with the TerraNova standardized test, a well‑established, nationally normed exam.
    • “To measure students’ levels of achievement at the end of the intervention, we administered the TerraNova standardized test. We chose TerraNova because it has well‑established and reputable psychometric qualities.” (p. 5)
    • Relevant Quotes: 1) “To measure students’ levels of achievement at the end of the intervention, we administered the TerraNova standardized test. We chose TerraNova because it has well‑established and reputable psychometric qualities.” (p. 5) 2) “Scores from the TerraNova Common Core assessment mathematics test were used as a primary outcome measure to provide data on student achievement in mathematics concepts, processes, and skills, and the test was administered at the end of students’ seventh-grade year.” (p. 6) Detailed Analysis: The study used a widely recognized, normed standardized exam (TerraNova) rather than a custom instrument. This ensures objective, comparable assessment across groups. Final sentence: Criterion E is met because a standard exam was employed.
  • T

    Term Duration

    • The intervention spanned a full academic year of seventh-grade instruction, exceeding the one-term minimum.
    • “We investigated what happened when teachers in Maine were newly trained to use ASSISTments and then deployed it with all their students throughout a full year of seventh-grade mathematics instruction.” (p. 3)
    • Relevant Quotes: 1) “We investigated what happened when teachers in Maine were newly trained to use ASSISTments and then deployed it with all their students throughout a full year of seventh-grade mathematics instruction.” (p. 3) 2) “As planned and executed, this experiment required that schools remain in the condition for 2 years and provide student data in the second year.” (p. 5) Detailed Analysis: The program was implemented over at least one full school year (and data were gathered in the second year), far surpassing the one-term requirement. Final sentence: Criterion T is met because the trial lasted a full academic year.
  • D

    Documented Control Group

    • The control group’s practices, demographics, and baseline scores are clearly documented for proper comparison.
    • “In the control group, teachers continued with their existing homework practices without any specific teacher training beyond that offered by their districts.” (p. 5)
    • Relevant Quotes: 1) “In the control group, teachers continued with their existing homework practices without any specific teacher training beyond that offered by their districts.” (p. 5) 2) “Control group teachers were not able to use ASSISTments with seventh-grade students.” (p. 5) 3) “The data included students’ achievement on the New England Common Assessment Program (NECAP) test for reading and mathematics. The data also included demographic indicators, such as gender, FRL status, and IEP status.” (p. 4) Detailed Analysis: The paper reports the control group’s condition (business-as-usual), collected baseline test scores, and student demographics, ensuring full documentation of the control group. Final sentence: Criterion D is met because the control group is well described.

• Level 2 Criteria

  • S

    School-level RCT

    • Entire schools (not just classes) were randomized, satisfying the school-level RCT requirement.
    • “We investigated this question through a randomized controlled experiment in which schools were assigned to either an intervention or a business-as-usual condition.” (p. 3)
    • Relevant Quotes: 1) “We investigated this question through a randomized controlled experiment in which schools were assigned to either an intervention or a business-as-usual condition.” (p. 3) 2) “Most schools were placed into pairs prior to randomization, and we proceeded with random assignment.” (p. 4) Detailed Analysis: Randomization occurred at the school level (43 schools), not merely within classes, ensuring a realistic, whole-school implementation scenario. Final sentence: Criterion S is met because the unit of randomization is the school.
  • I

    Independent Conduct

    • An independent research team evaluated the intervention, and no authors had a financial interest in ASSISTments.
    • “None of the authors has a financial interest in ASSISTments.” (p. 11)
    • Relevant Quotes: 1) “Worcester Polytechnic Institute provided the ASSISTments intervention—comprising the technology and closely related professional development—to schools, teachers, and students for this study.” (p. 3) 2) “None of the authors has a financial interest in ASSISTments.” (p. 11) Detailed Analysis: The tool was supplied by its creators (WPI), but the independent evaluation was done by researchers with no financial stake, ensuring unbiased data collection and analysis. Final sentence: Criterion I is met because evaluation was conducted independently.
  • Y

    Year Duration

    • The trial ran for a full academic year (with data from the second year cohort), meeting the one‑year duration requirement.
    • “Teachers used ASSISTments with a new cohort of seventh-grade students during the second school year. Scores from this second cohort of students were used for our analysis of the impact on student learning.” (p. 5)
    • Relevant Quotes: 1) “Teachers used ASSISTments with a new cohort of seventh-grade students during the second school year. Scores from this second cohort of students were used for our analysis of the impact on student learning.” (p. 5) 2) “As planned and executed, this experiment required that schools remain in the condition for 2 years and provide student data in the second year.” (p. 5) Detailed Analysis: Students experienced the intervention for an entire school year, satisfying the ERCT year‑long duration criterion. Final sentence: Criterion Y is met because the intervention covered a full year.
  • B

    Balanced Resources

    • Both groups followed identical homework policies and content; the ASSISTments tool and training were the core treatment.
    • “Both teacher groups were expected to assign and review mathematics homework for seventh-grade students in accordance with their local school policies.” (p. 5)
    • Relevant Quotes: 1) “Both teacher groups were expected to assign and review mathematics homework for seventh-grade students in accordance with their local school policies.” (p. 5) 2) “As previously described, the content of the homework in both groups came from the textbooks already in use at each school.” (p. 4) Detailed Analysis: The only difference was the ASSISTments platform and training, which were the intervention’s core. No extra time or budget was given aside from the intended treatment. Final sentence: Criterion B is met because resources and time were balanced.

• Level 3 Criteria

  • R

    Reproduced Results

    • An independent replication by another research team found similar positive results, confirming the original findings.
    • “Students at intervention schools performed significantly better than those in the comparison group on 8th grade End-of-Grade math tests one year after the program (p = 0.011, g = 0.10).” (Feng et al., 2023)
    • Relevant Quotes: 1) “Students at intervention schools performed significantly better than those in the comparison group on 8th grade End-of-Grade math tests one year after the program (p = 0.011, g = 0.10).” (Feng et al., 2023) Detailed Analysis: A large-scale replication in 63 North Carolina middle schools (with a new, independent research team) also found a significant improvement in math test scores for students using ASSISTments one year later. This confirms that the original positive effect was reproducible in a different population and setting, strengthening confidence in the results. Final sentence: Criterion R is met because an independent study validated the findings.
  • A

    All Exams

    • Only mathematics achievement was assessed; other subjects were not tested.
    • “Scores from the TerraNova Common Core assessment mathematics test were used as a primary outcome measure to provide data on student achievement in mathematics concepts, processes, and skills.” (p. 6)
    • Relevant Quotes: 1) “Scores from the TerraNova Common Core assessment mathematics test were used as a primary outcome measure to provide data on student achievement in mathematics concepts, processes, and skills.” (p. 6) Detailed Analysis: The intervention’s impact was measured only in math. No standardized tests in reading, science, or other core subjects were administered, failing the all‑subject exams requirement. Final sentence: Criterion A is not met because only one subject was assessed.
  • G

    Graduation Tracking

    • Students were originally tracked only through 7th grade; a separate follow-up measured outcomes at the end of 8th grade, but tracking did not extend to high school graduation.
    • “A follow-up study was designed to measure the long-term impact of ASSISTments on student learning in 2021, one year after the intervention was over.” (Feng et al., 2023)
    • Relevant Quotes: 1) “A follow-up study was designed to measure the long-term impact of ASSISTments on student learning in 2021, one year after the intervention was over.” (Feng et al., 2023) Detailed Analysis: This follow-up indicates the intervention’s impact persisted one year post-intervention (end of 8th grade). However, students were not tracked beyond middle school, meaning their high school outcomes (graduation or otherwise) remain unknown. Therefore, long-term tracking in the sense of following through graduation is absent. Final sentence: Criterion G is not met because participants were not tracked until graduation.
  • P

    Pre-Registered Protocol

    • The study was not pre-registered; no registry or preregistration statement is provided.
    • Relevant Quotes: None. Detailed Analysis: The paper does not mention any pre‑registration of hypotheses, methods, or analysis plans before data collection. Therefore, transparency via preregistration is lacking. Final sentence: Criterion P is not met because there is no preregistration.