Out-of-field and in-field teachers' professional learning gains on fostering understanding of multiplication: A randomized controlled trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was performed at the individual-teacher level within the same workshops, not at the class or school level, and no tutoring exception applies.
    • "Random assignment was administered within each of the three workshop sessions by random participant numbers leading participants to two different Lime-Survey activities for the systematization modes, Open Mode or Explicit Mode." (p. 15)
    • Relevant Quotes: 1) "The randomized controlled trial with n = 94 teachers investigates the extent to which out-of-field and in-field teachers' approximated practices can change in a PD session..." (Abstract, p. 1) 2) "For treating the research question, we conducted a randomized controlled trial as this methodology allows to test hypothesis on effects under highly controlled conditions." (p. 14) 3) "Random assignment was administered within each of the three workshop sessions by random participant numbers leading participants to two different Lime-Survey activities for the systematization modes, Open Mode or Explicit Mode." (p. 15) 4) "The independent variable varies two systematization modes, Explicit Mode ... and Open Mode..." (p. 14) Detailed Analysis: Criterion C requires randomisation at the class level or stronger (school level), unless the intervention is a one-to-one/tutoring intervention in which case student-level randomisation is acceptable. Here the unit of randomisation is the individual participating teacher: participants received "random participant numbers" routing them individually into either the Open Mode or the Explicit Mode within the very same workshop sessions. The two conditions therefore coexisted inside the same workshop groups, which is the within-group individual- level assignment the class-level criterion is designed to exclude. This is not a personal-tutoring intervention (it is a group professional-development session), so the tutoring exception does not apply. Criterion C is not met because randomisation was conducted at the individual-teacher level within shared workshops rather than at the class or school level.
  • E

    Exam-based Assessment

    • Outcomes were measured with researcher-developed pre/post vignettes, not with a standardised, widely recognised exam.
    • "Three pre-vignettes and post-vignettes on fostering students' understanding multiplication were given to the participants ... The vignette validity was assured in qualitative cognitive labs and expert judgments." (p. 14)
    • Relevant Quotes: 1) "Three pre-vignettes and post-vignettes on fostering students' understanding multiplication were given to the participants as integral parts of their PD experience in initial and concluding THINK activities, and random assignment was organized within the same workshops. The vignette validity was assured in qualitative cognitive labs and expert judgments." (p. 14) 2) "In Vignette 1 (Fig. 4), participants were asked to specify what understanding multiplication entails by writing an own explanation for the meaning of multiplication." (p. 14) 3) "The teachers' written responses to the open format Vignettes 1 and 2 ... were analyzed in a deductive-inductive code formation process ... They reached an interrater reliability of Cohens κ = 0.91 for Vignette 1 and Cohens κ = 0.83 for Vignette 2." (p. 16) Detailed Analysis: Criterion E requires the use of a standardised, widely recognised exam-based assessment that was not custom-built for the study. The outcome instruments here are three vignettes designed by the researchers, validated through their own cognitive labs and expert judgments, and scored with a study-specific coding scheme. These are bespoke measures of teachers' approximated practices, not a named, external standardised examination. Criterion E is not met because the dependent variable was captured through custom-designed vignettes rather than a standardised exam.
  • T

    Term Duration

    • The intervention and both measurements occurred inside a single ~100-120 min PD session, with the post-vignette taken after only 70 min, far short of one term.
    • "In this paper, we first publish data on in-field and out-of-field teachers' changes from the pre-vignette to the post-vignette after 70 min of PD session." (p. 18)
    • Relevant Quotes: 1) "In total, the intervention on fostering multiplication took 100 min including pre- and post-test (Fig. 3)." (p. 15) 2) "In the 10-min THINK Activity 5 (serving as post-vignette for the research), the participants are asked to 'think through' the same items as in Activity 1 to individually check how their approximated practices have changed." (p. 12) 3) "In this paper, we first publish data on in-field and out-of-field teachers' changes from the pre-vignette to the post-vignette after 70 min of PD session." (p. 18) 4) "The zoom-in into only 100 min of a PD session does not allow for measuring long-term learning gains..." (p. 25) Detailed Analysis: Criterion T requires the primary outcome to be measured at least one full academic term (roughly 3-4 months) after the intervention begins. In this study the pre-vignette, the intervention, and the post-vignette all occur within one 100-120 minute online PD session; the post-vignette is administered after about 70 minutes and the whole intervention lasts 100 minutes. The authors themselves note the study cannot measure long-term gains. The interval from intervention start to measurement is minutes, not a term. Criterion T is not met because outcomes were measured within the same short session, far below one academic term.
  • D

    Documented Control Group

    • The comparison (Open Mode) group and all subgroups are documented with demographics, sample sizes, and baseline pre-vignette performance.
    • "Table 1 documents the teachers' backgrounds for the whole sample and the subsamples." (p. 15)
    • Relevant Quotes: 1) "The study sample comprised all participants who completed the pre- and post-vignettes and indicated to be in-field teacher with a mathematics teaching certificate (n = 69.1%) or out-of-field teacher (n = 30.1%) ... These 94 teachers reported on average 8.1 years of teaching experience (SD = 8.9). 46% have encountered the Mastering Math curriculum material before." (p. 15) 2) "Table 1 documents the teachers' backgrounds for the whole sample and the subsamples." (p. 15) 3) Table 1 values: "Teachers in Open Mode: n 48, Teaching certificate in mathematics 68.8%, Prior encounter of curriculum material 46.7%, Years of teaching experience 10.3 (10.2); Teachers in Explicit Mode: n 46, 69.6%, 45.7%, 5.9 (6.9)." (Table 1, p. 15) 4) "In the open communicative systematization mode (Open Mode), participants are invited to discuss among themselves in breakout rooms about main insights they have gained (without moderation of a PD facilitator)." (p. 12) Detailed Analysis: This is a two-arm RCT (Open Mode vs Explicit Mode) with no untreated pure control; the Open Mode functions as the comparison condition against the Explicit Mode. Criterion D asks that this comparison group be well documented with demographics, baseline data, and the conditions it received. Table 1 reports sample sizes, the share holding a mathematics teaching certificate, prior curriculum encounter, and years of teaching experience for both randomised groups, and the pre-vignette figures (Figs. 5-7) report baseline performance. The treatment received by the comparison group (open breakout-room discussion during the systematization phase) is also described. Criterion D is met because the comparison (Open Mode) group and subsamples are documented with demographics, size, and baseline performance.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was at the individual-teacher level, not at the level of whole schools or implementing units.
    • "Random assignment was administered within each of the three workshop sessions by random participant numbers leading participants to two different Lime-Survey activities..." (p. 15)
    • Relevant Quotes: 1) "Random assignment was administered within each of the three workshop sessions by random participant numbers leading participants to two different Lime-Survey activities for the systematization modes, Open Mode or Explicit Mode." (p. 15) 2) "The participants were teachers in Grade 5 who volunteered to participate in a one-year PD program ... and came to a 120-min online PD session that was administered in video conferences on three afternoons after school." (p. 15) Detailed Analysis: Criterion S requires randomisation among whole schools (or the institutional unit implementing the intervention). The participants here are individual volunteer teachers randomised one-by-one within online workshops; no schools or institutional units were randomised. The randomised unit is the individual teacher. Criterion S is not met because assignment was at the individual-teacher level rather than the school level.
  • I

    Independent Conduct

    • The authors designed the Mastering Math PD program and also conducted, coded, and analysed the trial, with no independent third-party evaluator.
    • "This study belongs to the research project MuM video PD (BMBF grant No. 01JD2001A to S. Prediger ...). It draws upon the Mastering Math program, conducted within the German Center for Mathematics Teacher Education (DZLM)..." (p. 27)
    • Relevant Quotes: 1) "In this paper, we zoom into a one-year professional development (PD) program and investigate the design of the one PD session developed to promote teachers' professional learning..." (p. 2) 2) "This study belongs to the research project MuM video PD (BMBF grant No. 01JD2001A to S. Prediger by National Ministry of Education and Research). It draws upon the Mastering Math program, conducted within the German Center for Mathematics Teacher Education (DZLM)..." (p. 27) 3) "In Step 3, all written answers were rated by two coders (both blind to the treatment, the second author and an additional coder who received a coding manual and four-hour training on coding)." (p. 16) 4) "Both authors contributed equally to the research and the paper." (Author contributions, p. 27) Detailed Analysis: Criterion I requires the evaluation to be conducted independently of the intervention designers. The same author team (Prediger and Wischgoll) designed the Mastering Math PD program and its micro design, ran the PD session, coded the data (one of the two coders is the second author), and analysed the results. Although coders were blind to the treatment condition, blinding to condition is not independence from the designers; no external evaluation body conducted the study. Criterion I is not met because the intervention designers also carried out and analysed the trial without independent third-party oversight.
  • Y

    Year Duration

    • Because the term-duration criterion T is not met (outcomes measured within a single session), the stronger year duration criterion cannot be met.
    • "The zoom-in into only 100 min of a PD session does not allow for measuring long-term learning gains..." (p. 25)
    • Relevant Quotes: 1) "In total, the intervention on fostering multiplication took 100 min including pre- and post-test (Fig. 3)." (p. 15) 2) "In this paper, we first publish data on in-field and out-of-field teachers' changes from the pre-vignette to the post-vignette after 70 min of PD session." (p. 18) 3) "The zoom-in into only 100 min of a PD session does not allow for measuring long-term learning gains; these were already shown in earlier studies..." (p. 25) Detailed Analysis: Criterion Y requires outcomes to be measured at least ~75% of an academic year after the intervention begins, and it is explicitly conditioned on Criterion T being met. The measured intervention and outcomes occur within one 100-120 minute session, so neither T nor Y can be satisfied. Criterion Y is not met because the tracking interval was a single session, and Criterion T is already not met.
  • B

    Balanced Control Group

    • Both randomised conditions received the identical session structure and an equal 30-min systematization phase, differing only in mode (discussion vs. expert video), so time and resources were balanced.
    • "Both PD conditions had the same 70-min inquiry phase with THINK activities ... and 20-min transfer phase ..., but differed in the 30-min systematization phase." (p. 10)
    • Relevant Quotes: 1) "Both PD conditions had the same 70-min inquiry phase with THINK activities (individual work)—PAIR (partner work exchanging the individual ideas in breakout rooms) and SHARE settings ..., and 20-min transfer phase ..., but differed in the 30-min systematization phase." (p. 10) 2) "In the open communicative systematization mode (Open Mode), participants are invited to discuss among themselves in breakout rooms ... In the explicit instructional systematization mode (Explicit Mode), participants are invited to watch a 10-min systematizing expert video ... and to write individual notes." (p. 12) Detailed Analysis: Following the prescribed decision tree, the first question is whether the intervention adds extra time or budget to one arm relative to the other. It does not: both arms share the same 70-min inquiry phase, the same 20-min transfer phase, and an equal 30-min systematization phase. The only difference is the mode of systematization (open breakout-room discussion versus a 10-min expert video plus note-taking), which is the treatment variable being tested, not an added block of time or resources. Time on task and overall educational input are therefore matched across the two randomised groups, and no meaningful resource imbalance exists. Criterion B is met because both conditions received equivalent time and resources, differing only in the systematization mode that constitutes the treatment contrast.

• Level 3 Criteria

  • R

    Reproduced

    • This is a first-of-its-kind study with no independent replication by a different research team.
    • "In this paper, we first publish data on in-field and out-of-field teachers' changes from the pre-vignette to the post-vignette after 70 min of PD session." (p. 18)
    • Relevant Quotes: 1) "While an earlier paper provided overall accumulated evidence that their pedagogical content knowledge on multiplication can be further developed (Wischgoll & Prediger, 2024), this paper for the first time unpacks the different practices and inter-group differences between two target groups." (p. 23) 2) "An earlier study (Wischgoll & Prediger, 2024) revealed that the explicit mode had more overall professional learning effects, without disentangling them for particular practices and specific target groups." (p. 13) Detailed Analysis: Criterion R requires independent replication of this specific study by a different research team, published in a peer-reviewed outlet. The paper presents novel, first-time findings and references only the same authors' own prior work (Wischgoll & Prediger, 2024; Prediger et al., 2023, 2025). Internet verification note: a targeted web search for independent replications was attempted, but this session's automated web-search quota was already exhausted, so no live search could be executed. Grounding on the paper's own text, the authors explicitly frame the results as being published "for the first time," and given the 2026 publication date an independent, peer-reviewed replication by a different team is not plausible; none is known. Criterion R is not met because no independent replication by a different team exists.
  • A

    All-subject Exams

    • Criterion E is not met, and only multiplication-related teacher practices were assessed, so all-subject exam coverage cannot be satisfied.
    • "Three pre-vignettes and post-vignettes on fostering students' understanding multiplication were given to the participants..." (p. 14)
    • Relevant Quotes: 1) "The dependent variables were participants' professional learning gains in practices of specifying, monitoring, and enhancing the understanding of multiplication." (p. 14) 2) "the vignettes only focus one particular meaning in one specific representations..." (p. 25) Detailed Analysis: Criterion A requires standardised exam-based measurement across all main subjects and explicitly fails if Criterion E is not met. Here Criterion E is not met (custom vignettes), and the outcome is confined to a single narrow topic, one meaning of multiplication, measured only through teacher-response vignettes. Criterion A is not met because Criterion E is not met and only a single, narrow content area was assessed.
  • G

    Graduation Tracking

    • Criterion Y is not met and there is no follow-up beyond the single session, let alone tracking to graduation.
    • "The zoom-in into only 100 min of a PD session does not allow for measuring long-term learning gains..." (p. 25)
    • Relevant Quotes: 1) "Of course, short-term professional learning gains are no sufficient indication for long-term changes of really enacted classroom practices (Franke et al., 2007)." (p. 23) 2) "The zoom-in into only 100 min of a PD session does not allow for measuring long-term learning gains; these were already shown in earlier studies..." (p. 25) Detailed Analysis: Criterion G requires tracking participants through graduation and is explicitly conditioned on Criterion Y being met. Y is not met, and the study measures only immediate within-session change with no longitudinal follow-up. Internet verification note: the standard for G allows graduation tracking to appear in later papers by the same authors; the subjects here are practising teachers in a PD session, so graduation tracking is not applicable, and no follow-up publication tracking these participants to any graduation point exists. (A live web search could not be run because the session's web-search quota was exhausted; this reasoning is grounded in the paper's explicit statements that only short-term gains were measured.) Criterion G is not met because there is no long-term or graduation tracking, and Criterion Y is already not met.
  • P

    Pre-Registered

    • The paper contains no pre-registration statement, registry ID, or registration date.
    • Relevant Quotes: 1) "For treating the research question, we conducted a randomized controlled trial as this methodology allows to test hypothesis on effects under highly controlled conditions." (p. 14) 2) "All statistical analyses were done with SPSS 29.0 and conducted on an alpha level of 0.05." (p. 17) Detailed Analysis: Criterion P requires a public pre-registration of the study protocol (hypotheses, methods, analyses) before data collection, with a verifiable registry link and date. The paper states hypotheses and analysis plans within the article but provides no reference to any registry (e.g., OSF, AEA, ClinicalTrials.gov, DRKS), no registration ID, and no registration date. Internet verification note: because the paper names no registry entry at all, there is no registration record to look up in any database; nothing could be confirmed online (and the session's web-search quota was in any case exhausted). Criterion P is not met because no pre-registration of the protocol is reported.