Impact Evaluation of Complementarities Between Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice, Maryland, 2018-2021 (ICPSR 38863)

Version Date: Dec 10, 2024

Cite this study |

Principal Investigator(s):
Rebecca Herman, RAND Corporation; Troy D. Smith, RAND Corporation

https://doi.org/10.3886/ICPSR38863.v1

Version V1

Slide tabs to view more

Summary

Across the United States (U.S.), school districts have grappled with how to create safe community- and achievement-oriented schools and how to ensure the necessary discipline is applied transparently, fairly, and without bias. Two programs that many schools have turned to in order to achieve these goals are Schoolwide Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice (RJ). PBIS is an evidence-driven schoolwide behavioral management approach that aims to outline clear expectations for students and to cultivate shared norms and practices across classrooms and school spaces. PBIS has become a popular approach in schools and districts: as of 2020; over 19,000 schools in the U.S. have implemented PBIS.

A second program, Restorative Justice (RJ), has grown in popularity in recent years. RJ typically focuses on restorative relationship building between affected parties, peaceful reconciliation, and non-punitive approaches to rectifying harm, using a structured circle discussion format. RJ schools use both community circles, designed to build a safe space for students and staff to share and listen to each other, and restorative circles, designed to share perspectives on and redress a behavioral issue.

Working with a large school district in a mid-Atlantic state, researchers set out to test whether these two programs substitute for or complement each other. In partnership with the school district researchers conducted two separate school-level randomized controlled trials (RCTs). The first RCT (RCT 1) sought to uncover the marginal impact of RJ by adding the program to a set of randomly selected schools that were already implementing PBIS. The second RCT (RCT 2) was designed to discover the impact of introducing both programs together into schools that had neither program at baseline. Researchers conducted student and staff surveys to collect measures of school climate, teacher logs to record program implementation, and researchers also received administrative data from the district on student test scores, teacher and student absences, student disciplinary infractions, and school costs.

There is growing evidence of the effectiveness of each of these programs in isolation. A recent meta-analysis of 32 experimental and quasi-experimental impact studies of PBIS found that PBIS reduced disciplinary exclusions and problem behavior and increased academic achievement. The findings were statistically significant and showed small to medium effect sizes. Individual studies have found that PBIS reduces the use of office disciplinary referrals and other exclusionary disciplinary measures (including the use of in-school and out-of-school suspensions), while improving student behavior and attitudes across school levels. Individual studies show variable--some statistically significant and some null--impacts on academic outcomes.

The empirical evidence on the effect of RJ in U.S. schools is more limited, with little rigorous casual evidence published to date. Based on patterns across rigorous and non-rigorous research, restorative justice is associated with decreases in suspension rates and disciplinary disparities, improved student behavior, and improved school climate and relationships.

Citation

Herman, Rebecca, and Smith, Troy D. Impact Evaluation of Complementarities Between Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice, Maryland, 2018-2021. Inter-university Consortium for Political and Social Research [distributor], 2024-12-10. https://doi.org/10.3886/ICPSR38863.v1

Export Citation:

  • RIS (generic format for RefWorks, EndNote, etc.)
  • EndNote

Funding

United States Department of Justice. Office of Justice Programs. National Institute of Justice (2017-CK-BX-0017)

Subject Terms

Geographic Coverage

Distributor(s)

Inter-university Consortium for Political and Social Research
Hide

Time Period(s)

2018-01-01 -- 2021-12-31 ((The randomization was conducted in February 2019. The implementation of the interventions spanned school years 2019-2020 and 2020-2021. Administrative data was obtained for school years 2018-2019, 2019-2020, and 2020-2021)

Date of Collection

2019-05-20 -- 2019-06-06 (Site Visits - Wave 1), 2021-03-22 -- 2021-05-26 (Site Visits - Wave 3), 2019-06-01 -- 2021-05-31 (Teacher Logs - Monthly), 2020-03-02 -- 2020-03-26 (Staff survey - Wave 2), 2019-10-21 -- 2019-11-07 (Student survey - Wave 1), 2021-03-02
Hide

Study Purpose

The goal of the study was to hypothesize that the Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice (RJ) programs would be complementary, with the two being more effective than either program alone at improving school safety and discipline, and that the prosocial approach shared by both programs would reduce racial disparities in disciplinary responses. The formal research questions were as follows: 1. Do RJ and PBIS+RJ affect behavioral outcomes (i.e., disciplinary referrals, bullying, harassment, feeling isolated, social skills)? 2. Do RJ and PBIS+RJ reduce use of exclusionary practices (such as suspensions and expulsions)? 3. Do RJ and PBIS+RJ affect the disparities between African American, Hispanic/Latino, and White and Asian students on indicators of problem behavior, such as suspension rates? 4. Do RJ and PBIS+RJ affect students' attendance or academic outcomes? 5. Is PBIS+RJ more effective on the outcome measures than PBIS alone or neither program?

For both randomized controlled trials 1 (RCT 1) and randomized controlled trials 2 (RCT 2), researchers believed that the intervention (adding RJ to PBIS for RCT 1 and introducing RJ+PBIS for RCT 2) would positively affect behavioral outcomes and would reduce exclusionary practices and disparities in exclusionary practices. Researchers did not believe that there would be negative on academic outcomes and they believed that student and teacher attendance would increase. Although the RCTs were not set up to directly evaluate adding both programs to adding only one program (for example, as if all treatment conditions had been in a single RCT), researchers believed that the effect sizes in RCT 2 would be larger than the effect sizes in RCT 1 (i.e., that going from zero programs to two programs would have greater impact than going from one program to two programs).

Study Design

The school district informed schools of the study in school year (SY) 2018-2019 and invited schools to volunteer to take part. Schools were told that they would be randomly assigned to treatment or control groups and that district personnel would introduce Restorative Justice (RJ) or RJ+ Positive Behavioral Interventions and Supports (PBIS) into treatment schools over the following two academic years and control schools at the end of the two-year intervention. The latter was intended to keep the control schools engaged in study data collection during the period in which they did not receive interventions. Because schools were allowed to select into the experiment, it is likely that the schools that volunteered were more interested in positive behavioral interventions. Results should be interpreted as applying to such interested schools and may not be characteristic of all schools in the district.

Interested schools were then divided into two groups: those that were already implementing PBIS and those that were not (no elementary or middle schools in the district were implementing RJ at the time). The first group consisted of 23 schools (18 elementary schools and 5 middle schools), of which 12 were randomized into treatment (9 elementary schools and 3 middle schools) and the rest into the control condition. The second group consisted of 21 schools (15 elementary schools and 6 middle schools), of which 11 were randomized into treatment (8 elementary schools and 3 middle schools) and the rest into the control condition. The randomization was conducted in February 2019 to allow training for the treatment schools to start in the summer before the 2019 - 2020 academic year (SY 2019-2020). One treatment elementary school in randomized controlled trial 2 (RCT 2) dropped out of the experiment at the beginning of the 2019 - 2020 school year.

The final sample of 43 excludes one treatment elementary school in RCT 2 that dropped out of the experiment at the beginning of the 2019-2020 school year.

Time Method

Longitudinal

Universe

Student and staff in 43 elementary and middle schools in a United States school district.

Unit(s) of Observation

School, Staff, Student

Data Type(s)

Mode of Data Collection

Description of Variables

The study includes five datasets (3 administrative datasets and 2 survey datasets) and a total of 115 variables with unique IDs: School ID, Student ID, and Staff ID. The administrative data files had asked about the average teacher salary, average per pupil expenditure - Federal, if there was any suspension or expulsion, and Measures of Academic Progress (MAP) Reading score. The survey data files asked the respondents to describe if students get in fights, if students spread mean rumors or lies about others on social media or the internet, if they participated in meetings designed to address a problem or disagreement, the frequency of physical conflicts among students and if parents treat staff with respect.

The datasets also included additional demographic variables such as; male, minority, teachers, elementary school among others.

Response Rates

The study originally randomized 44 schools into treatment and control conditions. One treatment elementary school in randomized controlled trial 2 (RCT 2) dropped out of the experiment at the beginning of the 2019-2020 school year, leaving 43 schools in the final sample.

  • Administrative data: Administrative data was obtained for all students, staff, and schools for school years 2018-2019, 2019-2020, and 2020-2021.

  • Student survey: The student survey was administered to students in grades 5 and above. Active parental consent was required for student participation. The survey administration took place during classroom time: students with parental consent were given the choice to assent and to take the survey in English or Spanish. The logistics of seeking active parental consent and administering the survey in each school limited the response rates, which were 22.2 percent in Wave 1, 18.7 percent in Wave 2, and 17.5 percent in Wave 3.

  • Staff survey: Invitations to the staff survey were sent out by e-mail to all employed staff. Response rates were 49.1 percent for Wave 1, 32.9 percent for Wave 2, and 35.7 percent for Wave 3.

  • Site visits: Site visits were conducted at three times over the timespan of the study in nine study schools: five schools implementing both Restorative Justice (RJ) and Positive Behavioral Interventions and Supports (PBIS), two schools implementing PBIS only, and two schools implementing neither. Four were middle schools and five were elementary schools.

  • Teacher logs: A total of 2 teachers monthly per study school were randomly selected to fill out each monthly log. The total was increased to 4 per month starting in December 2020. The average response rate was 27.4 percent.

    • Presence of Common Scales

    Various Likert-type scales were used.

    Hide

    Original Release Date

    2024-12-10

    Version History

    2024-12-10 ICPSR data undergo a confidentiality review and are altered when necessary to limit the risk of disclosure. ICPSR also routinely creates ready-to-go data files along with setups in the major statistical software formats as well as standard codebooks to accompany the data. In addition to these procedures, ICPSR performed the following processing steps for this data collection:

    • Checked for undocumented or out-of-range codes.

    Hide

    Notes

    • The public-use data files in this collection are available for access by the general public. Access does not require affiliation with an ICPSR member institution.