Abstract
In secondary education, many students have difficulties planning their schoolwork. These difficulties may not only lead to short-term consequences such as lower grades, but also to long-term psychosocial, professional and financial challenges. To support students with planning problems, we developed a school-based program (Planning of Schoolwork (PLOS)) using an iterative development approach. PLOS includes two phases: PLOS-basic, a six-lesson preventive program with in-class skills training and PLOS-extra, a six-session, individual, modular skills training for students who still need support after PLOS-basic. This cluster randomized controlled trial compares the short- and long-term effects of PLOS-extra to practice-as-usual (PAU) with 213 first year students (aged 11-14, 78.9% male) who still had planning problems after PLOS-basic. Primary outcomes were homework problems and planning skills. Secondary outcomes included school motivation, classroom behavior, attention-deficit/hyperactivity disorder (ADHD) behaviors and school grades. We found no significant short- or long-term between-group differences on either the primary or the secondary outcome measures. Some significant improvements over time were present for both the PLOS-extra and the PAU group. The lack of effects of PLOS-extra may be attributed to a natural improvement of planning skills, the moderate severity of planning problems in our sample or the design of the program. Furthermore, the schools faced challenges in securing the necessary resources to implement the program effectively. Although PLOS-extra was developed in collaboration with end-users, these challenges demonstrate the complexity of developing training programs that are feasible within the educational context. School professionals and policymakers must decide whether to invest time, effort and resources into implementing evidence-based planning programs or in developing and testing new programs that offer support throughout daily academic life.
Full
Article
ERCT Criteria Breakdown
-
Level 1 Criteria
-
C
Class-level RCT
- The study randomized at the school level, satisfying the ERCT class-level-or-higher randomization requirement.
- "Schools were randomized over the experimental (PLOS-extra) and control group (PAU only; see Fig. 1)." (p. 4)
Relevant Quotes:
1) "Schools were randomized over the experimental (PLOS-extra) and control group (PAU only; see Fig. 1)." (p. 4)
2) "Cluster randomization was performed using a computer-generated list of random numbers applied to schools with strata on school size and school educational level to obtain equal numbers of participants in each group." (p. 4)
Detailed Analysis:
Criterion C requires randomization at the class level or higher (school level also satisfies this requirement). The paper states that schools were randomized between PLOS-extra and PAU, and describes a computer-generated randomization procedure applied to schools.
Final sentence explaining if criterion C is met/not met because the study randomized at the school level, which is stronger than the required class-level randomization.
-
E
Exam-based Assessment
- Outcomes rely on internal school grades and questionnaires, not a standardized exam-based assessment.
- "School grades were measured using the grades of the students over the complete schoolyear of the core subjects Mathematics, Dutch and English (that are given at each educational level and are mandatory by the Dutch Government) separately." (p. 8)
Relevant Quotes:
1) "School grades were measured using the grades of the students over the complete schoolyear of the core subjects Mathematics, Dutch and English (that are given at each educational level and are mandatory by the Dutch Government) separately." (p. 8)
2) "Twelve classes (of five different schools) worked with other grading systems that did not use numeric grades (in a scale from 0-10)." (p. 8)
Detailed Analysis:
Criterion E requires a standardized exam-based assessment that is widely recognized and comparable across participants. The paper measures academic performance using schools' own grading records, and explicitly notes heterogeneity in grading systems that required translation into numeric grades.
Final sentence explaining if criterion E is met/not met because the study used non-standardized internal grades (and questionnaires) rather than a single standardized external exam.
-
T
Term Duration
- The interval from intervention start after the January pre-test to the May/June post-test exceeds one academic term.
- "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
Relevant Quotes:
1) "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
2) "After the pre-test (January), students with planning problems were identified and students within the experimental PLOS-extra group were offered to receive PLOS-extra via an additional information letter." (p. 5)
Detailed Analysis:
Criterion T requires outcomes to be measured at least one academic term after the intervention begins. The procedures indicate that students were identified after the January pre-test and then began the PLOS-extra phase; the post-test occurred in May/June. This is roughly 4-6 months after January, which is at least one term.
Final sentence explaining if criterion T is met/not met because the post-test in May/June occurred more than one term after the January pre-test and the start of the PLOS-extra phase.
-
D
Documented Control Group
- The paper documents the PAU control group size, characteristics, and what support it could receive.
- "Table 2 shows the characteristics of the 213 students who were included the intent-to-treat analyses, of which 110 were in the experimental PLOS-extra group and 103 were in the PAU only group." (p. 9)
Relevant Quotes:
1) "Table 2 shows the characteristics of the 213 students who were included the intent-to-treat analyses, of which 110 were in the experimental PLOS-extra group and 103 were in the PAU only group." (p. 9)
2) "Participants in the PAU only group did not receive PLOS-extra but could receive support the school would normally provide or would be initiated outside of school, if available." (p. 7)
3) "Of the participants in the PAU only group, 22.4% indicated that they had received extra support within the school (homework support, tutoring in a specific course and planning skills support) and 7.8% of the participants indicated receiving extra support outside of school (homework support, mental health care support and tutoring in a specific course)." (p. 9)
Detailed Analysis:
Criterion D requires a well-documented control group (who they are, baseline characteristics, and what they received). The paper reports the PAU group size and provides baseline characteristics in Table 2, and it describes the PAU condition and quantifies additional support received in each group.
Final sentence explaining if criterion D is met/not met because the control group is clearly described in both tabular form and text, including what PAU support could occur.
-
Level 2 Criteria
-
S
School-level RCT
- Schools were the unit of randomization, meeting the school-level RCT criterion.
- "Schools were randomized over the experimental (PLOS-extra) and control group (PAU only; see Fig. 1)." (p. 4)
Relevant Quotes:
1) "Schools were randomized over the experimental (PLOS-extra) and control group (PAU only; see Fig. 1)." (p. 4)
2) "Eventually, fifteen schools were randomized to the PLOS-extra group and ten schools were randomized to the PAU only group." (p. 4)
Detailed Analysis:
Criterion S requires randomization among schools (not only among classes or students). The paper explicitly states that schools were randomized and reports the number of schools assigned to each arm.
Final sentence explaining if criterion S is met/not met because the unit of randomization was the school.
-
I
Independent Conduct
- The intervention was evaluated with substantial involvement of the intervention developers and author-led trainer training.
- "Bianca Boyer is co-developer and author of the intervention manuals ‘Plan My Life’, ‘Solution Focused Treatment’ and ‘My Sleep Plan’ and receives royalties for the sales of the interventions." (p. 1)
Relevant Quotes:
1) "Bianca Boyer is co-developer and author of the intervention manuals ‘Plan My Life’, ‘Solution Focused Treatment’ and ‘My Sleep Plan’ and receives royalties for the sales of the interventions." (p. 1)
2) "Further, she is co-developer and author of the intervention manuals ‘Plan My Life’ and ‘Solution Focused Treatment’ but does not receive royalties for the sales of the interventions." (p. 1)
3) "PLOS-extra trainers were trained in providing PLOS-extra by one of the authors (B.B., health care psychologist and author of PLOS) in a two-hour online training." (p. 5)
Detailed Analysis:
Criterion I requires that the evaluation be conducted independently from the intervention designers. The paper discloses that one author (Boyer) is a co-developer of intervention manuals and receives royalties, and that trainers were trained by an author described as "author of PLOS". These statements indicate direct involvement of intervention developers in implementation support and, by implication, in the evaluation process.
Final sentence explaining if criterion I is met/not met because the intervention developers were involved in trainer training and the author team includes co-developers, so the study is not clearly independent.
-
Y
Year Duration
- Outcomes were tracked from the December/January pre-test to an October/November follow-up, spanning roughly 9-10 months.
- "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
Relevant Quotes:
1) "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
2) "Data were collected over two school years (school year 1: 2021/22, school year 2: 2022/23). During both school years, we used the same cluster randomized controlled trial design." (p. 4)
Detailed Analysis:
Criterion Y requires outcome measurement at least one academic year after the intervention begins (typically about 9-10 months). The reported schedule includes a pre-test in December/January and a follow-up in October/November, which spans about 9-11 months and covers an academic-year-length interval.
Final sentence explaining if criterion Y is met/not met because the study includes an October/November follow-up roughly 9-10 months after the December/January pre-test period.
-
B
Balanced Control Group
- The extra time and staffing are integral to the intervention being tested (PLOS-extra), so PAU as the control is acceptable under the ERCT criterion B exception.
- "PLOS-extra (Boyer et al., 2021) is a modular skills training program that consists of six individual meetings, each lasting approximately 20-40 minutes during or after school." (p. 5)
Relevant Quotes:
1) "PLOS-extra (Boyer et al., 2021) is a modular skills training program that consists of six individual meetings, each lasting approximately 20-40 minutes during or after school." (p. 5)
2) "Participants in the PAU only group did not receive PLOS-extra but could receive support the school would normally provide or would be initiated outside of school, if available." (p. 7)
3) "This cluster randomized controlled trial compares the short- and long-term effects of PLOS-extra to practice-as-usual (PAU) with 213 first year students (aged 11-14, 78.9% male) who still had planning problems after PLOS-basic." (p. 1)
Detailed Analysis:
Criterion B asks whether additional resources (time, staffing, materials) are balanced between intervention and control, unless the additional resources are explicitly the treatment variable being tested. Here, the intervention is PLOS-extra itself, defined as six individual meetings delivered by a school mental health worker or trained staff. This implies additional staff time relative to business-as-usual.
The control condition is explicitly practice-as-usual and does not include PLOS-extra sessions. This is an imbalance in resources, but it is integral to what PLOS-extra is. The study's stated purpose is to compare the effects of providing this extra, structured skills training against PAU, so the additional time and staffing are the core treatment package rather than a supplementary add-on.
Final sentence explaining if criterion B is met/not met because the intervention explicitly tests the provision of PLOS-extra sessions as the treatment package against business-as-usual.
-
Level 3 Criteria
-
R
Reproduced
- No independent replication of this specific PLOS-extra effectiveness trial was located.
Relevant Quotes:
1) "For that reason, the new Dutch school-based program ‘Planning of Schoolwork’ (Wolters et al., 2021b; Boyer et al., 2021) was developed to support students in secondary education with planning problems." (p. 2)
Detailed Analysis:
Criterion R requires an independent replication by a different research team, in a different context, published in a peer-reviewed journal. The paper presents PLOS-extra as a new program and does not cite any independent replication of this specific trial. Targeted searches for an external replication of PLOS-extra did not identify a peer-reviewed replication study.
Final sentence explaining if criterion R is met/not met because no independent peer-reviewed replication of the PLOS-extra trial was found.
-
A
All-subject Exams
- Criterion A is not met because criterion E (standardized exam-based assessment) is not met.
Relevant Quotes:
1) "School grades were measured using the grades of the students over the complete schoolyear of the core subjects Mathematics, Dutch and English (that are given at each educational level and are mandatory by the Dutch Government) separately." (p. 8)
Detailed Analysis:
Criterion A requires standardized exam-based assessment across all main subjects, and the ERCT rules specify that if criterion E is not met, criterion A cannot be met either. Because the study uses internal grades rather than standardized exams, criterion A fails.
Final sentence explaining if criterion A is met/not met because it depends on criterion E, which is not met.
-
G
Graduation Tracking
- The trial followed students for six months, not through graduation, and no graduation follow-up paper was located.
- "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
Relevant Quotes:
1) "We conducted three measurements in both school years in which participating students, parents and mentor-teachers completed a questionnaire: 1) pre-test (December/January) 2) post-test (May/June) and 3) 6 months follow-up (October/November; only students and their parents; see Fig. 1)." (p. 4)
2) "This cluster randomized controlled trial compares the short- and long-term effects of PLOS-extra to practice-as-usual (PAU) with 213 first year students (aged 11-14, 78.9% male) who still had planning problems after PLOS-basic." (p. 1)
Detailed Analysis:
Criterion G requires tracking the cohort until graduation from the educational stage. The participants are first-year secondary school students, and the paper reports follow-up only through an October/November measurement described as a "6 months follow-up". This stops far short of secondary school graduation.
A targeted search for subsequent publications by the same authors tracking this cohort through graduation did not identify a peer-reviewed graduation follow-up report.
Final sentence explaining if criterion G is met/not met because the study stops at a six-month follow-up and provides no evidence of tracking the cohort to graduation.
-
P
Pre-Registered
- The paper reports prospective preregistration in REES before data collection, but the registry entry date could not be independently verified without login access.
- "This study's design was preregistered prospectively at Registry of Efficacy and Effectiveness Studies (REES) before data were collected (ID 9680.1v2)." (p. 5)
Relevant Quotes:
1) "This study's design was preregistered prospectively at Registry of Efficacy and Effectiveness Studies (REES) before data were collected (ID 9680.1v2)." (p. 5)
2) "At the beginning of the school year (September) participating school staff sent all first year students and their parents an information letter that described the aim and procedure of the study and the students and their parents were asked to give informed consent for data collection." (p. 5)
Detailed Analysis:
Criterion P requires preregistration of the study protocol before data collection begins. The paper explicitly states that the design was preregistered prospectively in REES and provides a registry ID. The procedures describe that schools began requesting consent for data collection at the beginning of the school year (September), with pre-test in December/January.
An attempt to verify the REES registry entry date directly was not possible because the REES site requires sign-in. Therefore, the timing is supported by the paper's explicit statement but not independently confirmed from the registry metadata.
Final sentence explaining if criterion P is met/not met because the paper states prospective REES preregistration before data collection, although the registry timestamp could not be verified without access.
Request an Update or Contact Us
Are you the author of this study? Let us know if you have any questions or updates.
