Abstract
While leveraging parents has the potential to increase student performance, programs that do so are often costly to implement or they target younger children. We partner text‐messaging technology with school information systems to automate the gathering and provision of information to parents at scale. In a field experiment across 22 middle and high schools, we used this technology to send automated text‐message alerts to parents about their child’s missed assignments, grades and class absences. We pre‐specified five primary outcomes. The intervention reduces course failures by 38% and increases class attendance by 17%. Students are more likely to be retained in the district. The positive effects are particularly large for students with below‐average GPA and students in high school. There are no effects on standardized test scores however. We randomly chose either the mother or the father to receive the alerts, but there were no differential effects across these subgroups. As in previous research, the intervention appears to change parents’ beliefs about their child’s performance and increases parent monitoring. Our results show that this type of automated technology can improve student effort relatively cheaply and at scale.
Full
Article
ERCT Criteria Breakdown
-
Level 1 Criteria
-
C
Class-level RCT
- The study randomised treatment at the grade‐by‐school level, satisfying the class‐level RCT requirement.
- “Random assignment was at the school‐by‐grade level to minimize the potential for spillovers into the control group.” (p. 9)
Relevant Quotes:
1) “Random assignment was at the school‐by‐grade level to minimize the potential for spillovers into the control group.” (p. 9)
Detailed Analysis:
The authors state that entire grade cohorts within each school were randomized, not individual students. This clustering at the grade‐by‐school level prevents contamination across treatment and control within the same classroom, satisfying the class‐level RCT requirement.
Final sentence explaining if criterion C is met because the unit of randomisation is at or above the class level.
-
E
Exam-based Assessment
- They used the Smarter Balanced standardized exams for Math and ELA as outcome measures.
- “The standardized test scores are from the Smarter Balanced assessment, which is aligned to the Common Core.” (p. 12)
Relevant Quotes:
1) “The standardized test scores are from the Smarter Balanced assessment, which is aligned to the Common Core.” (p. 12)
2) “We received scaled standardized test scores for Math and ELA for 2015 and 2016 examinations.” (p. 12)
Detailed Analysis:
The study uses the Smarter Balanced assessment, a widely recognized, state‐wide standardized exam aligned to the Common Core, for Math and English Language Arts. This fulfills the Exam‐based Assessment criterion.
Final sentence explaining if criterion E is met because standardized exams were used.
-
T
Term Duration
- The intervention lasted from late October through May, satisfying the full academic term requirement.
- “The intervention ran between the end of October 2015 through the end of May when the school year was expected to conclude.” (p. 10)
Relevant Quotes:
1) “The intervention ran between the end of October 2015 through the end of May when the school year was expected to conclude.” (p. 10)
Detailed Analysis:
The text‐message intervention spanned the majority of the 2015–2016 academic year, covering at least one full term (~semester equivalent). This meets the Term Duration requirement.
Final sentence explaining if criterion T is met because the study covered a full academic term.
-
D
Documented Control Group
- Control group characteristics and communications are clearly documented in the methods and Table 1.
- “Parents in the control group received the default level of information that the schools and teachers provided. This included report cards that are sent home after each marking period every six to nine weeks along with parent‐teacher conferences and any phone calls home from teachers.” (p. 10)
Relevant Quotes:
1) “Parents in the control group received the default level of information that the schools and teachers provided. This included report cards that are sent home after each marking period every six to nine weeks along with parent‐teacher conferences and any phone calls home from teachers.” (p. 10)
2) “Table 1 presents baseline summary statistics for the control group, the difference in means from the treatment group and the p‐value showing the statistical significance of these differences.” (p. 13)
Detailed Analysis:
The authors provide a clear description of the control group’s communications, baseline demographics and performance in Table 1, allowing proper comparison. This satisfies the Documented Control Group requirement.
Final sentence explaining if criterion D is met because control‐group details are fully documented.
-
Level 2 Criteria
-
S
School-level RCT
- Randomisation occurred at the grade‐by‐school level rather than entire schools.
- “Random assignment was at the school‐by‐grade level to minimize the potential for spillovers into the control group.” (p. 9)
Relevant Quotes:
1) “Random assignment was at the school‐by‐grade level to minimize the potential for spillovers into the control group.” (p. 9)
Detailed Analysis:
Randomisation occurred at the grade‐by‐school level, not at the whole school level. Multiple grades within the same school could be split between treatment and control, so the School‐level RCT criterion is not met.
Final sentence explaining if criterion S is not met because randomisation was below the whole‐school level.
-
I
Independent Conduct
- The study was designed, implemented, and analyzed by the same team without external evaluation.
Relevant Quotes:
No quotes indicate use of an external, third‐party evaluator.
Detailed Analysis:
All aspects of design, implementation and analysis were conducted by the authors (and their LMS partner) with no independent evaluation reported. This introduces potential bias, violating the Independent Conduct requirement.
Final sentence explaining if criterion I is not met because no external evaluator was used.
-
Y
Year Duration
-
B
Balanced Resources
- No additional instructional time or budget was provided, only low‑cost informational text messages.
- “Despite sending more than 32,000 text messages, the total cost of all of these messages was approximately $63. The gradebook and personnel training cost an additional $7 dollars per student.” (p. 4)
Relevant Quotes:
1) “Despite sending more than 32,000 text messages, the total cost of all of these messages was approximately $63. The gradebook and personnel training cost an additional $7 dollars per student.” (p. 4)
Detailed Analysis:
The only extra input was low‐cost text messages and minimal training. No additional instructional time or budget was provided, so there is no imbalance of educational resources between treatment and control.
Final sentence explaining if criterion B is met because no extra resources were added beyond information delivery.
-
Level 3 Criteria
-
R
Reproduced Results
-
A
All Exams
- Only Math and ELA were assessed via standardized exams, failing to cover all core subjects.
- “The standardized test scores are from the Smarter Balanced assessment, which is aligned to the Common Core.” (p. 12)
Relevant Quotes:
1) “The standardized test scores are from the Smarter Balanced assessment, which is aligned to the Common Core. We received scaled standardized test scores for Math and ELA for 2015 and 2016 examinations.” (p. 12)
Detailed Analysis:
Only Math and English Language Arts were assessed via standardized exams. Other core subjects (e.g., science, social studies) were not measured, so the All‑subject Exams criterion is not met.
Final sentence explaining if criterion A is not met because not all subjects were assessed.
-
G
Graduation Tracking
- Participants were tracked only through the end of the school year, not until graduation.
Relevant Quotes:
No quotes indicate follow‑up beyond the end of the school year.
Detailed Analysis:
The study tracks participants only through May of the intervention year. No data are collected through graduation, so the Graduation Tracking criterion is not met.
Final sentence explaining if criterion G is not met because no tracking continued to graduation.
-
P
Pre-Registered Protocol
- The study’s analysis plan and outcomes were publicly pre-registered before data collection began.
- “We pre-registered our analysis plan, subgroups of interest, and primary outcomes prior to obtaining outcome data.” (footnote 1)
Relevant Quotes:
1) “We pre-registered our analysis plan, subgroups of interest, and primary outcomes prior to obtaining outcome data.”
Detailed Analysis:
The authors explicitly state that they pre-registered their analysis plan and specified primary outcomes before collecting the outcome data. This indicates that a study protocol was established in advance, satisfying the Pre-registered Protocol criterion.
Final sentence explaining if criterion P is met because the study was pre-registered before data collection.
Request an Update or Contact Us
Are you the author of this study? Let us know if you have any questions or updates.
