Outcome Evaluation of Tribes Learning Communities in California, 2007-2010 (ICPSR 32821)

Time Period(s) View help for Time Period(s)

Date of Collection View help for Date of Collection

Data Collection Notes View help for Data Collection Notes

Study Purpose View help for Study Purpose

The purpose of the study was to assess the effectiveness of a school-based violence prevention program known as Tribes in preventing youth violence.

Study Design View help for Study Design

Tribes Learning Communities is a whole classroom (often whole school) strategy and on-going group process that uses teaching practices and class structures to develop an environment that promotes positive student behaviors and learning. The study took place from spring 2007 through spring 2010 and utilized a cluster-randomized experimental trial to assess the effectiveness of the Tribes Program in preventing youth violence. The study was designed in two phases: Phase I examined the impacts after one academic year of exposure to Tribes. Phase II examined the potential sustained program impacts six months after student exposure to one full academic year of Tribes in third and fourth grades.

Phase I of the study was conducted over 18 months starting in spring 2007 for Cohort 1 and summer 2008 for Cohort 2. Prior to random assignment, Cohort 1 teachers were assessed during spring 2007 (summer 2008 for Cohort 2 teacher), and again, after program implementation, in spring 2008 (spring 2009 for Cohort 2 teachers). Baseline measures for students were collected at the same time. Tribes professional development was delivered during the summer and fall prior to the beginning of implementation.

Among students, observations or measurement points occurred before and after the implementation of the intervention, in spring 2007 (Cohort 1)/summer 2008 (Cohort 2) and spring 2008 (Cohort 1)/2009 (Cohort 2), for Tribes and treatment-as-usual conditions. In the final year (Phase II), students were assessed in winter to evaluate the degree to which programs impacts were apparent six months after exposure to the program.

Data was obtained directly from the record systems of the districts that participated in the study (Dataset 1, Student Archival Data n= 3,992). The data were collected from participating school districts for kindergarten through third grade students enrolled in spring of Year 0 (2006-2007 for Cohort 1 schools and 2007-2008 for Cohort 2 schools), first through fourth grade students enrolled in spring of Year 1 (2007-2008 for Cohort 1 and 2008-2009 for Cohort 2 schools), and second through fifth grade students enrolled in spring of Year 2 (2008-2009 for Cohort 1 schools and 2009-2010 for Cohort 2 schools).

Teacher-reported behavior checklist (Dataset 2, Teacher Checklist Data n= 2,624) were collected for students in kindergarten through third grade in the spring of Year 0 (baseline), for students in first through fourth grades in the spring of Year 1 (implementation), and students in fourth and fifth grades in the spring of Year 2 (follow up).

Parent-reported behavior checklists (Dataset 3, Parent Checklist Data n= 1,211) were collected from parents in the spring of Year 1 and Year 2. Checklists in four languages (English, Spanish, Chinese, and Tagalog) were used to collect the student behavior data from parents.

Evaluators conducted individual clinical style interviews with a randomly selected subset of a maximum three students from each third and fourth grade classroom at two time points in Year 1 (Dataset 4, Student Interview Data n= 140). Four vignettes about incidents of disruptive and antisocial behaviour that students typically encounter in clossrooms and schools were developed. Each 30-40 minute student interview consisted of two vignettes and were tape-recorded.

Teacher surveys (Dataset 5, Teacher Survey Data n= 213) were administered prior to, and at the end of the intervention (Phase I) to all treatment and control teachers.

Classroom observation data (Dataset 6, Classroom Observation Data n= 128) were collected at two time points in Year 1. The 30 minute observations were carried out in both treatment and control classrooms across all grades. Observers were not told whether the classroom they were observing was an intervention classroom and participating teachers were asked not to inform observers of their status. The observation protocol also included a brief 10 minute post-observation interview with the teacher, administered within 24 hours of the observation, to assess how typical the observed practices and behaviors were.

Teacher attendance data collected at the trainings (Dataset 7, Teacher Training Data n= 81) were used to describe the number of teachers who participated in the Tribes professional development as well as follow-up training or remote training in Year 1.

Data containing the rosters of students (Dataset 8, Student Roster Data n= 4,221) and teachers (Dataset 9, Teacher Roster Data n= 302) were obtained directly from the schools. The information was collected from the scools in the spring of Year 0 and in the fall of Year 1 and Year 2.

From these nine base data files researchers generated five analytic sample files: Student Impact Analysis Data (Dataset 10), Student Followup Impact Analysis Data (Dataset 11), Student Interview Analysis Data (Dataset 12), Teacher Impact Analysis Data (Dataset 13), and Classroom Observation Impact Analysis Data (Dataset 14).

Sample View help for Sample

The sample for Dataset 1 through Dataset 9 were selected as follows. Recruitment of districts and schools for the study began in February, 2006. The project targeted districts and schools that were ethnically and economically diverse, with staff who were interested in implementing Tribes and willing to support a randomized trial research design. The San Francisco Unified School District (SFUSD) responded postively to a solicitation to participate in the study. Seven SFUSD schools (Cohort 1) with nearly half the required classrooms entered the experiment in the Fall of 2007. For Cohort 2, which includes schools from the SFUSD as well as four other school districts, the recruitment strategy focused on schools instead of districts using a mass mailing to elementary schools in the larger Bay Area. Year-round schools were initially targeted, followed by public and charter schools. By the end of June 2008, an additional four schools had been recruited. However, the recruitment of teachers from those schools had to wait until staff returned in August 2008.

Presentations at staff meetings about the field trial, samples of the data collection instruments, and handouts detailing study requirements reinforced the necessity that those teachers assigned to the experimental group had to aviod sharing Tribes strategies with teachers assigned to the control group. Additional efforts were made to recruit teachers hired late within the SFUSD via personal contacts and phone calls, and with the assistance of the site coordinator. Of the 208 potential teachers in the eligiblity pool, 166 returned consent forms and 42 either directly or indirectly refused to participate. Of the teachers who did not return the consent form, follow-up efforts revealed that most of these teachers had either retired, moved to other positions in the district, went on maternity leave, or left the district altogether. Once signed consent were received from the teachers indicating their agreement to participate, they were randomized into either the Tribes experimental group or the control group. Teachers' classrooms, not students or schools, served as the unit of randomization.

The procedures used to secure parental permission for student study participation differed between Cohort 1 and Cohort 2 schools. For Cohort 1, institutional review board requirements called for written parental permission (active parental consent) for students to participate in the study. Despite repeated efforts significant difficulties were encountered in securing parental consent forms, with greater difficulty in low-performing schools with large percentages of ethnic minorities. Because of those difficulties, a formal exemption from institutional review was requested and approved for Cohort 2 schools. As a result, no consent documents were required for Cohort 2, although active consent was still obtained for students selected to participate in interviews. All first through fourth grade students in Cohort 2 schools and their parents were notified of the study and given the chance to opt out if they preferred by sending in a signed form asking that their child not participate (passive parental consent).

For Cohort 1, materials requesting parental consent were distributed to site coordinators at each school in spring 2007 (prior to the first implementation year). Site coordinators ensured that consent forms were distributed to all parents of students in kindergarten through third grade. After the consent materials were first distributed to parents, they were then redistributed periodically to those parents who had not yet returned consent forms. Subsequently, in fall 2007, materials requesting parental consent were distributed to parents of newly enrolled students in first through fourth grades and to parents who had not returned consent forms during the prior spring.

The analytic sample for the Student Impact Analysis Data (Dataset 10) consisted of 2,309 students in first through fourth grade with parental consent in participating teachers' classrooms in Year 1.

The analytic sample for the Student Followup Impact Analysis Data (Dataset 11) consisted of 661 of the participating Year 1 third and fourth grade students enrolled in a fourth grade control classroom or fifth grade classroom in Year 2.

The analytic sample for the Student Interview Impact Analysis (Dataset 12) included 134 students who participated in student interviews in Year 1.

The analytic sample for the Teacher Survey Impact Analysis (Dataset 13) consisted of 159 participating teachers who were randomized and had returned teacher surveys.

The analytic sample for the Classroom Observation Impact Analysis (Dataset 14) included 126 participating teachers who were randomized and taught in the classrooms where a classroom observation was conducted.

Time Method View help for Time Method

Universe View help for Universe

All Kindergarten through Fifth grade students and teachers in the San Francisco Unified School District between 2007 and 2010.

Unit(s) of Observation View help for Unit(s) of Observation

Data Type(s) View help for Data Type(s)

Mode of Data Collection View help for Mode of Data Collection

Description of Variables View help for Description of Variables

The student archival dataset (Dataset 1, Student Archival Data) contains information on language arts and mathematics test scale scores from the California Standards Test (CST), student demographic characteristics (grade, gender, ethnicity, student eligibility for free or reduced price lunch program, English learner status/English proficiency, parent education level, and student special education designation), student attendance (number of days student attended school and number of absences), and suspension data.

The teacher-reported checklist (Dataset 2, Teacher Checklist Data) included measures from the Achenbach Behavior Checklist (teacher form) (ABCL) and the Behavioral and Emotional Rating Scale (BERS-2). Teachers were asked to rate the individual student's social competencies, interactions, internal states, and classroom behavior. The checklist measured six constructs (Aggression, Hyperactivity, Bullying, Conduct problems, Defiance, and Violence) and assessed students on several behavior syndromes. The ABCL was used to assess aggressive behavior, rule-breaking behavior, social problems, and attention problems. The BERS-2 was used to assess positive student behaviors that might result from the implementation of Tribes and changes in classroom environments. BERS-2 asked teachers to rate individual student's strengths and competenecies in five domains: interpersonal strength, intrapersonal strength, school functioning, and affective strength.

The parent-reported checklist (Dataset 3, Parent Checklist Data) included the Achenbach Behavior Checklist (parent form) (ABCL) and the Behavioral and Emotional Rating Scale (BERS-2). The questions included in the parent-reported checklist were based on the same constructs used in the teacher-reported checklist including Aggression, Hyperactivity, Bullying, Conduct problems, Defiance, and Violence.

Each vignette (Dataset 4, Student Interview Data) asked students about the benefits and harm of the behaviors described in the story, the reasons they may occur, and what they think should be done about preventing and/or intervening in these incidents. The interviews also included questions about antisocial attitudes and behaviors including victimization and bullying, and what students have learned from Tribes.

Items on the teacher survey (Dataset 5, Teacher Survey Data) were used to assess the following outcomes: positive student behavior, student centered teaching practices, use of student reflection practices, student supportive learning practices, cooperative learning groups, and small group activities.

The classroom observations (Dataset 6, Classroom Observation Data) were used to assess whether the teacher provided opportunities to work in small groups, opportunities for collaboration, opportunities for students to reflect on what they had learned; that students were engaged with the teacher and their classmates, respected their classmates, and shared materials and helped their classmates. Small groups, collaboration and student reflection were measured with dichotomous indicators while student engagement, respect, and sharing were measured with multiple Likert items.

Teacher training data (Dataset 7, Teacher Training Data) includes information on the Tribes professional development training for teachers such as when the teacher received the training, the number of days of the offered training that the teacher received, and whether the teacher received the follow-up or remote training.

The student roster file (Dataset 8, Student Roster Data) includes the student study identification number, student demographic information (grade, gender, arbitrary district and school identification numbers), teacher study identification number, student consent status, and samples selected for student interview and parent survey administration.

The teacher roster file (Dataset 9, Teacher Roster Data) includes teacher study identification number, grade taught, arbitrary district and school identification numbers, treatment group assignment, and teacher consent status.

Response Rates View help for Response Rates

The response rate for Dataset 1, Student Archival Data, was 97 percent. The Teacher Checklist Data (Dataset 2) response rate was 94 percent. The Parent Checklist Data (Dataset 3) response rate was 78 percent. The response rate for Dataset 4, Student Interview Data, was 58 percent. The Teacher Survey Data (Dataset 5) response rate was 94 percent.

Presence of Common Scales View help for Presence of Common Scales

Student Archival Data (Dataset 1) includes the total scale scores for English language arts and Mathematics from the California Standards Tests (CST).

Teacher Checklist Data (Dataset 2) includes measures obtained from the Achenbach Behavior Checklist (teacher form) (ABCL) and the Behavioral and Emotional Rating Scale (BERS-2).

The Parent Checklist Data (Dataset 3) includes measures obtained from the Achenbach Behavior Checklist (parent form) (ABCL) and the Behavioral and Emotional Rating Scale (BERS-2).

Original Release Date View help for Original Release Date

2012-12-20

Version History View help for Version History

Weight View help for Weight

To reduce Cohort 1 non-response bias in the analysis, the Student Roster Data (Dataset 8) includes the variable "WEIGHT". Sucessfully recruited students were weighted by the inverse of the sampling rate in each classroom. For example, if eleven students in a classroom did not return consent forms, researchers randomly sampled seven of those eleven students for more intensive follow up. If four of the seven sampled students were sucessfully recruited, researchers weighted the four cases by the inverse of the within class sampling rate, or 11/7. Students within the classroom who had already returned consent forms received a weight of 1. Students in Cohort 2 received a weight of 1. The variable WEIGHT is also found in the three student level analysis files (Dataset 10, Student Impact Analysis Data, Dataset 11, Student Followup Impact Analysis Data, and Dataset 12, Student Interview Analysis Data).