Evaluating Observational Data Analyses: Confounding Control and Treatment Effect Heterogeneity [Methods Study], United States, 2013-2019 (ICPSR 39485)
Version Date: Sep 3, 2025 View help for published
Principal Investigator(s): View help for Principal Investigator(s)
Issa J. Dahabreh, Brown University
https://doi.org/10.3886/ICPSR39485.v1
Version V1
Summary View help for Summary
A randomized trial is one of the best ways to learn if one treatment works better than another. Randomized trials assign patients to different treatments by chance. But they are not always affordable, and they take a long time to complete.
When randomized trials aren't possible, researchers can use observational studies to learn how treatments work. In observational studies, researchers look at what happens when patients and their doctors choose the treatments. Traits such as age or health may affect treatment choices. These traits may also affect patients' responses to treatment, making it hard to know if the treatment or the traits affected the patients' responses.
Some study designs and statistical methods may help address this problem and make results from observational studies more useful. These methods can give researchers more data about whether treatments work and how the same treatment can affect groups of patients differently.
The research team conducted three studies to test different methods of designing and analyzing observational studies. They wanted to know if observational studies that used these methods produced results similar to randomized trials.
Citation View help for Citation
Export Citation:
Funding View help for Funding
Subject Terms View help for Subject Terms
Geographic Coverage View help for Geographic Coverage
Distributor(s) View help for Distributor(s)
Study Purpose View help for Study Purpose
To evaluate whether methods for estimating treatment effects and heterogeneity of treatment effects (HTE) from observational data produce results consistent with evidence from randomized trials. The study aimed to address the following research questions: (1) If state-of-the-science analysis methods are used in information-rich observational data, do observational data analyses emulating target trials produce marginal (population) effects consistent with prior randomized evidence and background knowledge? (2) Do observational analyses emulating target trials agree with randomized trials and background knowledge when assessing heterogeneity of treatment effects? (3) How do different methods for assessing effect heterogeneity perform in observational studies?
Study Design View help for Study Design
In this project, researchers conducted three studies to test different methods of designing and analyzing observational studies to estimate treatment effects and HTE. Treatment effects refer to differences in the outcome distributions when comparing two or more treatments applied to the same individuals. HTE means the treatment effect is not the same for all individuals.
In study 1, researchers compared treatment effects from observational studies with treatment effects from randomized trials of the same interventions. Using data from claims and electronic health records, researchers developed four observational studies to emulate one hypothetical randomized trial comparing the effectiveness of antihypertensive medicines. Researchers used multiple methods, including propensity score-based methods, to estimate treatment effects in each emulation. They then estimated treatment effects for antihypertensive medicines in a pooled analysis of 11 trials and compared results from the emulations with the pooled trial analysis.
In study 2, researchers evaluated a novel approach of baseline risk stratification to assess HTE. Researchers used data from 32 randomized trials and modeled the predicted outcome risk in each trial using covariates associated with the outcome. Based on the predicted outcome risk, researchers stratified the trial population into four strata of increasing predicted risk. Researchers then assessed HTE by comparing absolute and relative risk reduction across the four risk strata.
In study 3, researchers compared the performance of different HTE estimation methods using simulated and real data. Researchers simulated different treatment scenarios and patient subgroups. They assessed HTE using outcome regression, inverse probability weighting, and doubly robust methods.
Clinicians, patients, clinical practice guideline developers, regulatory authorities, and funders provided input on this study.
Data Source View help for Data Source
Study 1: randomized trials in public databases or provided by investigators; observational data from an integrated data set of claims, electronic health records, and registries
Study 2: randomized trials in public databases or provided by investigators
Study 3: simulated data, randomized trial and observational data
Notes
The public-use data files in this collection are available for access by the general public. Access does not require affiliation with an ICPSR member institution.
ICPSR usually offers files in multiple formats for researchers to be able to access data and documentation in formats that work well within their needs. If you have questions about the accessibility of materials distributed by ICPSR or require further assistance, please visit ICPSR’s Accessibility Center.