What else should I know about the MET Project?

thumbnail of study organization chart

The data collections available through the MET Longitudinal Database have been organized into multiple ICPSR studies in an attempt to provide clarity to how the many different data files relate to each other. There are data collections, and thus ICPSR studies, from three sample sets from within, or related to, the MET project: the full MET sample, which includes all MET teachers and students; the randomization sample, drawn from the full MET sample; and the district-wide census, from which the full MET sample was drawn. The data within the collections, and thus within the different ICPSR studies, are also made available at six different levels of analysis: student, teacher, class/section, school, video observation segment, and survey or measurement item-level.

Recruitment of Districts

MET researchers recruited districts into the study during the period of July - November 2009. The final selection of districts was based on a district's interest in the study, staff size sufficient to assure adequate numbers of participating teachers, central office support for the MET program, willingness and capacity to participate in all parts of the data collection process, and broader local political and union support for the project. At the end of recruitment, the following districts were selected for and participated in the study: Charlotte-Mecklenburg (NC) Schools, Dallas (TX) Independent School District, Denver (CO) Public Schools, Hillsborough County (FL) Public Schools, Memphis (TN) City Schools, and the New York City (NY) Department of Education.

Recruitment of Schools

Within each recruited district, certain schools were excluded from participation in the study including special education schools, alternative schools, community schools, autonomous dropout and pregnancy programs, returning education schools, and vocational schools that did not teach academic courses. Also excluded from the study were schools that employed team teaching or other structural features that made it impossible to assign responsibility for a student's learning to a single, specific teacher.

Recruitment of Teachers

Once a school principal agreed to participate in the study, all teachers assigned to teach MET Study focal grade/subject combinations were invited to participate in the study unless: (a) they were team teaching or looping, making it impossible to assign responsibility for the learning of a given student in a specific subject to that teacher; (b) the teacher indicated that he or she was not planning to stay in the same school and teach the same subject the following year; or (c) there were fewer than two other teachers with the same grade/subject teaching assignments. This last restriction was put into place to assure that each teacher could be put into an "exchange" group for random assignment of classes to teachers in Year Two of the study.

Realized Samples

In Year One of the study, a total of 2,741 teachers in 317 schools took part in the MET Study, distributed across grade/subject groupings as indicated in the table below. By contrast, the Year Two sample includes just 2,086 teachers in 310 schools. The table below shows the realized sample sizes for the MET teacher samples, for both years of the study. The Year One sample (left hand column) shows all teachers who participated in Year One of the study, regardless of their eligibility for the randomization that took place in Year Two. The Year Two sample (right hand column) shows the number of teachers who participated in Year Two of the study broken out by their randomization status.

Year One MET Teacher Sample vs. MET LDB Core Teacher Sample by Focal Grade/Subject
Full Sample
All Year One Teachers
(AY 2009-2010)
Core Study Sample
All Teachers Present in Year Two
(AY 2009-2010)
Randomized Non-Randomized
4th and 5th Grade English/Language Arts (ELA) 138 98 29
4th and 5th Grade Mathematics 102 67 31
4th and 5th Grade ELA and Mathematics 634 305 52
Grades 6-8 ELA 606 292 139
Grades 6-8 Mathematics 528 282 120
Grades 6-8 ELA and Mathematics 18 4 4
9th Grade Algebra I 233 116 44
9th Grade English 242 108 48
9th Grade Biology 240 103 60

Student achievement gains on state standardized tests, supplemental tests, and value-added measures

MET researchers used state test scores and administered supplemental assessment tests as one measure student learning. While the state tests are designed to measure how well students have learned by the state standards, supplemental tests tend to measure reasoning skills and conceptual understanding. From these, and other instruments described below, "value-added" measures of teacher effects on student learning were created.

Classroom observations

To see how well different classroom observation tools identify effective teaching, MET project researchers recorded four lessons each year in each participating teacher's classroom. These recording were then scored using one or more of the six protocols listed below:

  • Classroom Assessment Scoring System (CLASS), developed by Robert Pianta, University of Virginia
  • Framework for Teaching, developed by Charlotte Danielson
  • Mathematical Quality of Instruction (MQI), developed by Heather Hill, Harvard University, and Deborah Loewenberg Ball, University of Michigan
  • Protocol for Language Arts Teaching Observations (PLATO), developed by Pam Grossman, Stanford University
  • Quality Science Teaching (QST) Instrument, developed by Raymond Pecheone, Stanford University

Teachers' pedagogical content knowledge

In the second year of the project (2010-2011), participating teachers took assessments to measure their ability to choose appropriate strategies and to recognize and diagnose common student errors.

Student perceptions of the classroom instructional environment

All students in participating teachers' classrooms completed surveys about their experience in the classroom and their teachers' ability to engage them in the course material.

Teachers' perceptions of working conditions and support at their schools

All participating teachers completed surveys asking them about the quality of working conditions within their schools and the amount of instructional support they receive.

When schools joined the study during the 2009-10 school year, principals identified groups of teachers in which all teachers met the following criteria:

  1. They were teaching the same subject to students in the same grade (for example, teachers teaching math to 6th graders or English language arts to 8th graders or self-contained 4th grade classes).
  2. They had the necessary certification so they could all teach common classes.
  3. They were expected to teach the same subject to students in the same grade in the 2010-11 school year.

These groups of teachers were referred to as "exchange groups," and schools needed at least one exchange group with two or more teachers who agreed to enroll in the study to participate in the MET project.

The project requested scheduling information for 2,462 teachers from 865 exchange groups in 316 schools. The project created 668 randomization blocks from 619 exchange groups in 284 of the participating schools. The remaining schools' schedules did not permit randomly swapping rosters among any of MET project teachers or all its MET project teachers had left the school or the study.

From these randomization blocks, the project randomly assigned rosters to 1,591 teachers. (This includes 386 high school teachers and 24 teachers for whom rosters were later found to be invalid.) Seventy teachers were not eligible for randomization because they were not scheduled to teach the exchange group subject and grade level in 2010-11 or they decided not to participate in year 2 of the study. The remaining 281 teachers could not be randomized because they did not teach in a period with two or more teachers for exchanging rosters.

Video scoring was conducted in several phases. There was an initial summer pilot phase during which time a subset of 413 teachers with complete data had their 2,000 videos scored with the CLASS protocol. These videos are sometimes called the "Plan B" sample. This scoring occurred prior to use of the web-based coding interface. Phase 1 occurred once this interface was established and scored the same 2000 videos using the rest of the observation protocols. Phase 2 scoring occurred later and focused on scoring both years of videos from teachers that were successfully randomized in Year Two. Phase 3 consisted of scoring videos that were only 25 minutes long on the FFT protocol.

For more information on phases of video observation scoring see Chapter 10 of the User Guide.