Interconnecting Positive Behavioral Interventions and Supports (PBIS) and School Mental Health to Improve School Safety, South Carolina and Florida, 2013-2020 (ICPSR 37908)

Time Period(s) View help for Time Period(s)

Date of Collection View help for Date of Collection

Data Collection Notes View help for Data Collection Notes

Study Purpose View help for Study Purpose

There is growing momentum to improve school climate and safety by implementing comprehensive strategies that address the underlying causes of misbehavior, including exposure to violence and other mental health issues, and to replace punitive disciplinary practices with efforts to help students develop prosocial skills and behaviors. Two primary approaches include Positive Behavioral Interventions and Supports (PBIS) and School Mental Health (SMH).

PBIS is a holistic, multi-tiered, evidence-based approach for preventing and reducing aggression and other problem behavior in school through the implementation of universal prevention (Tier 1) for all children, targeted intervention (Tier 2) for children at risk or showing early signs of problems, and intensive interventions (Tier 3) for children and youth with more significant problems. Despite widespread adoption, in general, intervention effects of PBIS have been modest. Two key limitations are that the majority of PBIS schools struggle with intensive interventions at Tier 2 and Tier 3 and the emphasis is on behavior with limited attention on issues that can exacerbate problem behavior such as depression, anxiety, emotional dysregulation, and trauma. SMH services have been shown to significantly improve access to care and early identification and intervention, as well as improvements to student outcomes and positively influence school safety. However, SMH lacks an implementation structure resulting in students effectively needing to be at crisis level to be referred for services.

The Interconnected Systems Framework (ISF) addresses limitations of PBIS and SMH in addressing school safety by providing specific guidance on their systematic interconnection. Key components of ISF emphasize interdisciplinary collaboration, the functioning of teams, improving data-based decision making, and improving the selection and implementation of evidence-based practices. The purpose of this longitudinal study was to evaluate the contribution of the ISF in improving school safety, school climate, behavioral and discipline problems, and school outcomes in students. This randomized controlled study was the first experimental evaluation of the ISF's contributions to school and student safety and functioning above the effects of PBIS alone or PBIS and SMH clinicians operating in normative co-located fashion.

Study Design View help for Study Design

School-wide assessment of teacher/staff reports of school climate and safety and student records were collected for all students in grades K-5. Data were entered, scored, and maintained by the school year (June 15th to June 15th annually). Student-level analyses focused on students who entered the study at the end of 4th grade, and had post-assessment at the end of 5th grade, and follow-up assessments at the end of 6th grade.

For the 2016-2017 school year, there were 15,649 students (46.5 percent White) in schools across all three conditions and one combined condition. There were 3,773 students in Positive Behavioral Interventions and Supports (PBIS) only schools (49.3 percent White); 5,802 students in PBIS and School Mental Health (SMH) schools (36.2 percent White); 5,627 students in Interconnected Systems Framework (ISF) schools (57.6 percent White); and one school that changed from PBIS only in year 1 to PBIS+SMH in year 2 with 447 students (15.4 percent White). There were 4,789 4th and 5th grade students (49.34 percent White, 49.84 percent male). Of teachers/school staff who completed the universal teacher survey for the 2016-2017 school year, 91.32 percent were female (N=682), and of the 678 who indicated race, 79.02 percent were White.

For the 2017-2018 school year, there were 14,978 students (48.1 percent White) across all three conditions and one combined condition. There were 3,613 students in PBIS only schools (49.3 percent White); 5,842 students in PBIS+SMH schools (40.7 percent White); 5,072 students in ISF schools (58.6 percent White); and the one school that added SMH to PBIS in Year 2, with 451 students (17.8 percent White). There were 4,572 4th and 5th grade students (48.18 percent White, 50.26 percent male). Of teachers/school staff who completed the universal teacher survey for the 2017-2018 school year, 88.45 percent were female (N=708), and of the 698 who indicated race, 78.26 percent were White.

Faculty/staff measures: School staff surveys were collected during the 2015-2016 (pre-intervention), 2016-2017 (post-intervention 1), and 2017-2018 (post-intervention 2) school years. All school staff were invited to participate in the schoolwide surveys. Teacher perceptions of school climate were assessed using scales from the Authoritative School Climate Survey (ASCS), including teacher perceptions of fairness, student willingness to seek help, teacher respect for students, student affective engagement, student cognitive engagement, and extent of teasing and bullying. Teacher perceptions of school safety were measured using the Safety scale from the teacher version of the Effective School Battery (ESB).

Teacher ratings of selected students: Teachers completed the Strengths and Difficulties Questionnaire (SDQ) for youth aged 3-17 years, which assessed a range of emotional and behavioral problems (EB) and prosocial behavior. Teachers also completed the Physical Aggression subscale of the New York Teacher Rating Scale (NYTRS), developed for students in 1st - 10th grades for deeper assessment of disruptive behavior disorders. There were two cohorts of students selected, one cohort of students selected prior to ISF implementation (2015-2016) and one cohort selected after ISF implementation (2016-2017).

Student ratings: Student surveys were collected from two cohorts of students starting in 4th grade. The first cohort of students completed the surveys starting in the 2015-2016 school year, prior to the beginning of the data collection. The second cohort completed the surveys starting in 2016-2017, after intervention began. Student survey responses were collected from students in all 24 schools. Like the teacher version, the student SDQ assesses EB problems and prosocial behavior. In addition, students evaluated their satisfaction with the most recent mental health services received using the Youth Satisfaction Questionnaire (YSQ). Student perceptions of school climate were measured using scales from the ASCS, assessing perceptions of fairness and strictness of discipline, supportiveness of teachers, affective engagement, and the extent of bullying and teasing. Students also completed items from the Exposure to Violence Screening Measure (EVSM), reflecting recent exposure to violence (e.g., robbery, assault, shooting).

School record data: School records were retrieved on all students in grades K-5 for three years prior to implementation, as well as each year of the study. Variables collected included demographic data, attendance, behavior, educational placement/disability status, and course performance/grades.

PBIS team variables: All PBIS teams (average size of 8 members) for the three conditions were asked to record the following during each team meeting: names/disciplines of participants, meeting duration, and which students received Tier 2 or Tier 3 services and the types of services they received. At the beginning and end of the study intervention years, teams completed the Benchmark for Advanced Tiers (BAT) and the Tiered Fidelity Inventory (TFI) to measure the quality of team processes and delivery and refinement of evidence-based practices in each tier.

Fidelity and cost data: In ISF schools, implementation fidelity was evaluated through data collected for each team meeting and use of the ISF Implementation Inventory (ISF-II) in the fall and spring of intervention years. The cost data reflects the costs of implementation for the academic year June 2017-2018 and are based on actual expenditures. Cost data were assembled from study financial records and interviews with study investigators.

Sample View help for Sample

Two large school districts in the Southeastern United States (South Carolina and Florida) were recruited for this study, with 12 elementary schools from each participating. Participating schools were selected by meeting the following criteria: served students kindergarten through 5th grade, implemented PBIS with fidelity the previous school year, and did not have SMH presence prior to the study. In each district, four schools were randomly assigned to one of three conditions: PBIS only, PBIS+SMH, or ISF. The intervention phase was implemented for two consecutive years, involving two cohorts of students. All students attending schools receiving an intervention condition received the intervention assigned.

Time Method View help for Time Method

Universe View help for Universe

Elementary and middle school aged youth in South Carolina and Florida and school staff members who support them.

Unit(s) of Observation View help for Unit(s) of Observation

Data Source View help for Data Source

Data Type(s) View help for Data Type(s)

Mode of Data Collection View help for Mode of Data Collection

Response Rates View help for Response Rates

Due to the systemic level intervention, which is considered a routine educational practice, all students who required an intervention condition received the one assigned to their respective schools. The subset of students completing questionnaires were recruited by an opt-out procedure. The opt-out rate ranged between 4 percent and 10 percent depending on cohort and school district.

For Cohort 1 students who completed the baseline assessment in Spring 2016, 17 percent did not complete measures in Spring 2017 (post-assessment), and 22 percent did not complete measures in Spring 2018 (follow-up assessment). For Cohort 2 students who completed the baseline assessment in Spring 2017, 26 percent did not complete measures in Spring 2018, and 22 percent did not complete measures in Spring 2019.

Presence of Common Scales View help for Presence of Common Scales

• Authoritative School Climate Survey (ASCS)
• Behavioral and Emotional Screening System (BESS)
• Benchmark for Advanced Tiers (BAT)
• Bully-Victimization Scale (BVS)
• Effective School Battery (ESB): Teacher version, Safety scale
• Exposure to Violence Screening Measure (EVSM)
• Interconnected Systems Framework Implementation Inventory (ISF-II)
• New York Teacher Rating Scale (NYTRS): Physical Aggression subscale
• Receipt and Satisfaction with Services Scale
• Racial/Ethnic Bullying Scale
• Strengths and Difficulties Questionnaire (SDQ)
• Tiered Fidelity Inventory (TFI)
• Youth Satisfaction Questionnaire (YSQ)

Original Release Date View help for Original Release Date

2023-11-16

Version History View help for Version History

Weight View help for Weight

Not applicable.