Panacea or Poison: Can Propensity Score Modeling (PSM) Methods Replicate the Results from Randomized Control Trials (RCTs)?, United States, 1983-2013 (ICPSR 37291)
Version Date: Aug 14, 2023 View help for published
Principal Investigator(s): View help for Principal Investigator(s)
Christopher M. Campbell, Portland State University;
Ryan M. Labrecque, University of Central Florida
https://doi.org/10.3886/ICPSR37291.v1
Version V1
Summary View help for Summary
With the growing popularity, technological ease of using propensity score modeling (PSM), and the concern over its reliability and validity among scholars and practitioners, the researchers aimed to answer whether PSM methods can replicate the results from randomized controlled trials (RCTs). In this secondary data analysis, the researchers gathered the datasets of 10 publicly available and restricted RCT studies from the National Archive of Criminal Justice Data (NACJD), introduced an artificial selection bias into the treatment groups of these investigations, and then used each PSM technique to remove this selection bias. The team then compared the results generated from the PSM methods to those derived from the original RCT experiments, and meta-analyzed the findings across all studies to reveal the true reliability and validity of PSM in relation to RCTs using criminal justice data.
For each study used in this analysis, the researchers created SPSS syntax for variable recodes and artificial bias creation and a codebook with original study items, recoded variables, and analytic variables. (In one study, two RCTs were conducted and thus two sets of syntax and codebooks were created.) Seven text files contain the Stata and R code used to run each PSM technique. These materials have been zipped into a package and are available for restricted download. Please refer to the ICPSR README for more information.
Citation View help for Citation
Export Citation:
Funding View help for Funding
Subject Terms View help for Subject Terms
Geographic Coverage View help for Geographic Coverage
Smallest Geographic Unit View help for Smallest Geographic Unit
None
Restrictions View help for Restrictions
Access to these data is restricted. Users interested in obtaining these data must complete a Restricted Data Use Agreement, specify the reason for the request, and obtain IRB approval or notice of exemption for their research.
Distributor(s) View help for Distributor(s)
Time Period(s) View help for Time Period(s)
Date of Collection View help for Date of Collection
Data Collection Notes View help for Data Collection Notes
-
For the 10 datasets upon which this analysis is based, please refer to each study's ICPSR page (see Data Source for a complete listing) for data, documentation, and more information on background, study design, methodology, sampling, and variables.
Study Purpose View help for Study Purpose
The purpose of the study was to determine whether results obtained via randomized controlled trial can be replicated by using propensity score matching in criminal justice research studies.
Study Design View help for Study Design
A total of seven propensity score matching (PSM) techniques were used in this analysis due to their common usage in criminal justice research: one-to-one matching (with and without caliper), one-to-many matching (with and without caliper), inverse probability of treatment weighting (IPTW), stratified weighting scheme, and optimal pairs matching.
In order to test the different PSM techniques, selection bias was systematically introduced into the treatment groups by identifying the most critical measures in each study and those that best predicted the likelihood of being in the treatment group, constructing an additive scale from variables that successfully predicted treatment group membership, and selecting only the treatment cases above the scale's mean. This process was done in a blind manner, meaning that the measures used to create the bias were unknown to the researcher who conditioned the propensity score.
The biased and PSM treatment and control groups were compared to each other as well as the RCT using six measures of balance (e.g., percentage of covariates with statistically significant differences and the standardized percent bias). For each dataset, a common metric effect size for all dependent variables in the original RCT and for the seven PSM techniques after removing the artificial selection bias. The PSM and RCT results were compared in relation to their effect sizes and direction (i.e., positive or negative). The researchers calculated the difference in the estimated outcomes between the RCT and PSM methodologies and averaged across the total number of outcomes included. The overall effect size for the difference between each PSM and RCT approach was calculated using a random-effects model meta-analysis.
Sample View help for Sample
In order to be included in this evaluation, studies needed to use a random assignment procedure in assigning cases to groups and have at least 130 cases in each condition group (based on the threshold needed to properly detect effect size with enough power). A systematic search of NACJD identified 46 potential studies, from which only nine studies of different types met the sample size requirements. A quasi-experimental study that identified comparison group participants prior to program rollout was added to raise the number of studies to 10.
Time Method View help for Time Method
Universe View help for Universe
Criminology and criminal justice research studies that involved a randomized controlled trial.
Unit(s) of Observation View help for Unit(s) of Observation
Data Source View help for Data Source
Portland [Oregon] Domestic Violence Experiment, 1996-1997 (ICPSR 3353)
Police Departments' Use of Lethality Assessments: An Experimental Evaluation (ICPSR 34975)
Domestic Violence Experiment in King's County (Brooklyn), New York, 1995-1997 (ICPSR 4307)
Reducing Fear of Crime: Program Evaluation Surveys in Newark and Houston, 1983-1984 (ICPSR 8496)
Data Type(s) View help for Data Type(s)
Description of Variables View help for Description of Variables
For each dataset, measures were created for predictors for treatment membership group and artificial selection bias (if a case had a significant bias relative to comparison cases).
Response Rates View help for Response Rates
Not applicable.
Presence of Common Scales View help for Presence of Common Scales
Not applicable.
HideNotes
These data are part of NACJD's Fast Track Release and are distributed as they were received from the data depositor. The files have been zipped by NACJD for release, but not checked or processed except for the removal of direct identifiers. Users should refer to the accompanying readme file for a brief description of the files available with this collection and consult the investigator(s) if further information is needed.
The public-use data files in this collection are available for access by the general public. Access does not require affiliation with an ICPSR member institution.
One or more files in this data collection have special restrictions. Restricted data files are not available for direct download from the website; click on the Restricted Data button to learn more.
This dataset is maintained and distributed by the National Archive of Criminal Justice Data (NACJD), the criminal justice archive within ICPSR. NACJD is primarily sponsored by three agencies within the U.S. Department of Justice: the Bureau of Justice Statistics, the National Institute of Justice, and the Office of Juvenile Justice and Delinquency Prevention.