A Two-Step Framework for Validating Causal Effect Estimates.
Humans
Observational Studies as Topic
/ methods
Randomized Controlled Trials as Topic
/ methods
Causality
Computer Simulation
Confounding Factors, Epidemiologic
Research Design
Registries
/ statistics & numerical data
Reproducibility of Results
Bias
Selection Bias
Data Interpretation, Statistical
Pharmacoepidemiology
/ methods
causal estimates
sampling mechanism
treatment assignment mechanism
validation
Journal
Pharmacoepidemiology and drug safety
ISSN: 1099-1557
Titre abrégé: Pharmacoepidemiol Drug Saf
Pays: England
ID NLM: 9208369
Informations de publication
Date de publication:
Sep 2024
Sep 2024
Historique:
revised:
25
06
2024
received:
18
11
2023
accepted:
26
06
2024
medline:
10
9
2024
pubmed:
10
9
2024
entrez:
10
9
2024
Statut:
ppublish
Résumé
Comparing causal effect estimates obtained using observational data to those obtained from the gold standard (i.e., randomized controlled trials [RCTs]) helps assess the validity of these estimates. However, comparisons are challenging due to differences between observational data and RCT generated data. The unknown treatment assignment mechanism in the observational data and varying sampling mechanisms between the RCT and the observational data can lead to confounding and sampling bias, respectively. The objective of this study is to propose a two-step framework to validate causal effect estimates obtained from observational data by adjusting for both mechanisms. An estimator of causal effects related to the two mechanisms is constructed. A two-step framework for comparing causal effect estimates is derived from the estimator. An R package RCTrep is developed to implement the framework in practice. A simulation study is conducted to show that using our framework observational data can produce causal effect estimates similar to those of an RCT. A real-world application of the framework to validate treatment effects of adjuvant chemotherapy obtained from registry data is demonstrated. This study constructs a framework for comparing causal effect estimates between observational data and RCT data, facilitating the assessment of the validity of causal effect estimates obtained from observational data.
Sections du résumé
BACKGROUND
BACKGROUND
Comparing causal effect estimates obtained using observational data to those obtained from the gold standard (i.e., randomized controlled trials [RCTs]) helps assess the validity of these estimates. However, comparisons are challenging due to differences between observational data and RCT generated data. The unknown treatment assignment mechanism in the observational data and varying sampling mechanisms between the RCT and the observational data can lead to confounding and sampling bias, respectively.
AIMS
OBJECTIVE
The objective of this study is to propose a two-step framework to validate causal effect estimates obtained from observational data by adjusting for both mechanisms.
MATERIALS AND METHODS
METHODS
An estimator of causal effects related to the two mechanisms is constructed. A two-step framework for comparing causal effect estimates is derived from the estimator. An R package RCTrep is developed to implement the framework in practice.
RESULTS
RESULTS
A simulation study is conducted to show that using our framework observational data can produce causal effect estimates similar to those of an RCT. A real-world application of the framework to validate treatment effects of adjuvant chemotherapy obtained from registry data is demonstrated.
CONCLUSION
CONCLUSIONS
This study constructs a framework for comparing causal effect estimates between observational data and RCT data, facilitating the assessment of the validity of causal effect estimates obtained from observational data.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
e5873Informations de copyright
© 2024 The Author(s). Pharmacoepidemiology and Drug Safety published by John Wiley & Sons Ltd.
Références
M. A. Hernán and J. M. Robins, “Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available,” American Journal of Epidemiology 183, no. 8 (2016): 758–764.
S. V. Wang, S. Schneeweiss, J. M. Franklin, et al., “Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials,” Journal of the American Medical Association 329, no. 16 (2023): 1376–1385.
B. K. Beaulieu‐Jones, S. G. Finlayson, W. Yuan, et al., “Examining the Use of Real‐World Evidence in the Regulatory Process,” Clinical Pharmacology & Therapeutics 107, no. 4 (2020): 843–852.
J. M. Franklin and S. Schneeweiss, “When and How can Real World Data Analyses Substitute for Randomized Controlled Trials?” Clinical Pharmacology & Therapeutics 102, no. 6 (2017): 924–933.
R. Saesen, M. van Hemelrijck, J. Bogaerts, et al., “Defining the Role of Real‐World Data in Cancer Clinical Research: The Position of the European Organisation for Research and Treatment of Cancer,” European Journal of Cancer 1, no. 186 (2023): 52–61.
J. M. Franklin, E. Patorno, R. J. Desai, et al., “Emulating Randomized Clinical Trials With Nonrandomized Real‐World Evidence Studies: First Results From the RCT DUPLICATE Initiative,” Circulation 143, no. 10 (2021): 1002–1013.
G. Imbens and K. Menzel, “A Causal Bootstrap.” (National Bureau of Economic Research, Cambridge, MA, July 16, 2018).
Y. Chen, P. Li, and C. Wu, “Doubly Robust Inference With Nonprobability Survey Samples,” Journal of the American Statistical Association 115, no. 532 (2020): 2011–2021.
I. J. Dahabreh, S. E. Robertson, J. A. Steingrimsson, E. A. Stuart, and M. A. Hernan, “Extending Inferences From a Randomized Trial to a New Target Population,” Statistics in Medicine 39, no. 14 (2020): 1999–2014.
M. A. Hernán, “A Definition of Causal Effect for Epidemiological Research,” Journal of Epidemiology & Community Health 58, no. 4 (2004): 265–271.
C. R. Lesko, A. L. Buchanan, D. Westreich, J. K. Edwards, M. G. Hudgens, and S. R. Cole, “Generalizing Study Results: A Potential Outcomes Perspective,” Epidemiology 28, no. 4 (2017): 553–561.
B. A. Dickerman, X. Garcı́a‐Albéniz, R. W. Logan, et al., “Avoidable Flaws in Observational Analyses: An Application to Statins and Cancer,” Nature Medicine 25, no. 10 (2019): 1601–1606.
J. Concato, N. Shah, and R. I. Horwitz, “Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs,” New England Journal of Medicine 342, no. 25 (2000): 1887–1892.
S. P. Forbes and I. J. Dahabreh, “Benchmarking Observational Analyses Against Randomized Trials: A Review of Studies Assessing Propensity Score Methods,” Journal of General Internal Medicine 35 (2020): 1396–1404.
I. J. Dahabreh, R. C. Sheldrick, J. K. Paulus, et al., “Do Observational Studies Using Propensity Score Methods Agree With Randomized Trials? A Systematic Comparison of Studies on Acute Coronary Syndromes,” European Heart Journal 33, no. 15 (2012): 1893–1901.
L. G. Hemkens, D. G. Contopoulos‐Ioannidis, and J. P. Ioannidis, “Agreement of Treatment Effects for Mortality From Routinely Collected Data and Subsequent Randomized Trials: Meta‐Epidemiological Survey,” BMJ 352, no. 8044 (2016): i493.
S. R. Cole and E. A. Stuart, “Generalizing Evidence From Randomized Clinical Trials to Target Populations: The ACTG 320 Trial,” American Journal of Epidemiology 172, no. 1 (2010): 107–115.
E. A. Stuart, S. R. Cole, C. P. Bradshaw, and P. J. Leaf, “The Use of Propensity Scores to Assess the Generalizability of Results From Randomized Trials,” Journal of the Royal Statistical Society: Series A (Statistics in Society) 174, no. 2 (2011): 369–386.
B. Colnet, I. Mayer, G. Chen, et al., “Causal Inference Methods for Combining Randomized Trials and Observational Studies: A Review,” 2020, arXiv Preprint arXiv:2011.08047.
P. Hünermund and E. Bareinboim, “Causal Inference and Data Fusion in Econometrics,” The Econometrics Journal 10 (2023): utad008.
I. Degtiar and S. Rose, “A Review of Generalizability and Transportability,” 2021, arXiv Preprint arXiv:2102.11904.
K. P. Josey, S. A. Berkowitz, D. Ghosh, and S. Raghavan, “Transporting Experimental Results With Entropy Balancing,” Statistics in Medicine 40, no. 19 (2021): 4310–4326.
I. J. Dahabreh, A. Matthews, J. A. Steingrimsson, D. O. Scharfstein, and E. A. Stuart, “Using Trial and Observational Data to Assess Effectiveness: Trial Emulation, Transportability, Benchmarking, and Joint Analysis,” Epidemiologic Reviews (2023): mxac011.
M. A. Hernán, W. Wang, and D. E. Leaf, “Target Trial Emulation: A Framework for Causal Inference From Observational Data,” Journal of the American Medical Association 328, no. 24 (2022): 2446–2447.
G. W. Imbens and D. B. Rubin, Causal Inference in Statistics, Social, and Biomedical Sciences (Boca Raton, FL: Cambridge University Press, 2015).
S. L. Lohr, Sampling: Design and Analysis (Boca Raton, FL: CRC Press, 2021).
P. R. Rosenbaum and D. B. Rubin, “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika 70, no. 1 (1983): 41–55.
Y. Tillé and M. Wilhelm, “Probability Sampling Designs: Principles for Choice of Design and Balancing,” Statistical Science 32, no. 2 (2017): 176–189.
L. Lei and E. J. Candès, “Conformal Inference of Counterfactuals and Individual Treatment Effects,” Journal of the Royal Statistical Society, Series B: Statistical Methodology 83, no. 5 (2021): 911–938.
J. Hoogland, J. Int Hout, M. Belias, et al., “A Tutorial on Individualized Treatment Effect Prediction From Randomized Trials With a Binary Endpoint,” Statistics in Medicine 40, no. 26 (2021): 5961–5981.
A. Swaminathan, “Counterfactual Evaluation and Learning From Logged User Feedback” (PhD Thesis, University of Cornell, 2017).
J. M. Snowden, S. Rose, and K. M. Mortimer, “Implementation of G‐Computation on a Simulated Data set: Demonstration of a Causal Inference Technique,” American Journal of Epidemiology 173, no. 7 (2011): 731–738.
H. Bang and J. M. Robins, “Doubly Robust Estimation in Missing Data and Causal Inference Models,” Biometrics 61, no. 4 (2005): 962–973.
Z. Tan, “Bounded, Efficient and Doubly Robust Estimation With Inverse Weighting,” Biometrika 97, no. 3 (2010): 661–682.
M. J. van der Laan and S. Rose, Targeted Learning (New York, NY: Springer, 2011).
J. Robins, M. Sued, Q. Lei‐Gomez, and A. Rotnitzky, “Comment: Performance of Double‐Robust Estimators When “Inverse Probability” Weights are Highly Variable,” Statistical Science 22, no. 4 (2007): 544–559.
S. R. Seaman and S. Vansteelandt, “Introduction to Double Robust Methods for Incomplete Data,” Statistical Science: A Review Journal of the Institute of Mathematical Statistics 33, no. 2 (2018): 184–197.
A. Tsiatis, Semiparametric Theory and Missing Data (Boca Raton, FL: Springer Science & Business Media, 2007).
M. A. Brookhart, S. Schneeweiss, K. J. Rothman, R. J. Glynn, J. Avorn, and T. Stürmer, “Variable Selection for Propensity Score Models,” American Journal of Epidemiology 163, no. 12 (2006): 1149–1156.
R. Prentice, “Use of the Logistic Model in Retrospective Studies,” Biometrics 32 (1976): 599–606.
V. Dorie, J. Hill, U. Shalit, M. Scott, and D. Cervone, “Automated Versus Do‐It‐Yourself Methods for Causal Inference: Lessons Learned From a Data Analysis Competition,” Statistical Science 34, no. 1 (2019): 43–68.
P. R. Hahn, J. Murray, and C. M. Carvalho, “Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects,” Bayesian Analysis 15, no. 3 (2017): 965–1056.
T. Wendling, K. Jung, A. Callahan, A. Schuler, N. H. Shah, and B. Gallego, “Comparing Methods for Estimation of Heterogeneous Treatment Effects Using Observational Data From Health Care Databases,” Statistics in Medicine 37, no. 23 (2018): 3309–3324.
J. L. Hill, “Bayesian Nonparametric Modeling for Causal Inference,” Journal of Computational and Graphical Statistics 20, no. 1 (2011): 217–240.
P. N. Zivich and A. Breskin, “Machine Learning for Causal Inference: On the use of Cross‐Fit Estimators,” Epidemiology 32, no. 3 (2021): 393–401.
A. I. Naimi, A. E. Mishler, and E. H. Kennedy, “Challenges in Obtaining Valid Causal Effect Estimates With Machine Learning Algorithms,” American Journal of Epidemiology 192, no. 9 (2023): 1536–1544.
V. Chernozhukov, D. Chetverikov, M. Demirer, et al., “Double/Debiased Machine Learning for Treatment and Structural Parameters,” The Econometrics Journal 21, no. 1 (2018): 1–68.
Y. Zhong, E. H. Kennedy, L. M. Bodnar, and A. I. Naimi, “AIPW: An R Package for Augmented Inverse Probability–Weighted Estimation of Average Causal Effects,” American Journal of Epidemiology 190, no. 12 (2021): 2690–2699.
A. Chattopadhyay, C. H. Hase, and J. R. Zubizarreta, “Balancing vs Modeling Approaches to Weighting in Practice,” Statistics in Medicine 39, no. 24 (2020): 3227–3254.
A. L. Buchanan, M. G. Hudgens, S. R. Cole, et al., “Generalizing Evidence From Randomized Trials Using Inverse Probability of Sampling Weights,” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181, no. 4 (2018): 1193–1209.
E. A. Stuart, “Matching Methods for Causal Inference: A Review and a Look Forward,” Statistical Science: A Review Journal of the Institute of Mathematical Statistics 25, no. 1 (2010): 1–21.
K. Hirano, G. W. Imbens, and G. Ridder, “Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score,” Econometrica 71, no. 4 (2003): 1161–1189.
F. Li, K. L. Morgan, and A. M. Zaslavsky, “Balancing Covariates via Propensity Score Weighting,” Journal of the American Statistical Association 113, no. 521 (2018): 390–400.
L. Shen, G. Geleijnse, and M. Kaptein, “RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects.” RCTrep.
T. André, C. Boni, M. Navarro, et al., “Improved Overall Survival With Oxaliplatin, Fluorouracil, and Leucovorin as Adjuvant Treatment in Stage II or III Colon Cancer in the MOSAIC Trial,” Journal of Clinical Oncology 27, no. 19 (2009): 3109–3116.
L. Shen, A. van Gestel, P. Prinsen, et al., “Value of Real‐World Evidence for Treatment Selection: A Case Study in Colon Cancer,” JCO Clinical Cancer Informatics 8 (2024): e2300186.
A. Chatton, F. Le Borgne, C. Leyrat, et al., “G‐Computation, Propensity Score‐Based Methods, and Targeted Maximum Likelihood Estimator for Causal Inference With Different Covariates Sets: A Comparative Simulation Study,” Scientific Reports 10, no. 1 (2020): 1–3.
J. L. Hong, M. Webster‐Clark, M. Jonsson Funk, et al., “Comparison of Methods to Generalize Randomized Clinical Trial Results Without Individual‐Level Data for the Target Population,” American Journal of Epidemiology 188, no. 2 (2019): 426–437.
B. Colnet, J. Josse, G. Varoquaux, and E. Scornet, “Reweighting the RCT for Generalization: Finite Sample Error and Variable Selection.” 2022.
C. O'Muircheartaigh and L. V. Hedges, “Generalizing From Unrepresentative Experiments: A Stratified Propensity Score Approach,” Journal of the Royal Statistical Society: Series C: Applied Statistics 63, no. 2 (2014): 195–210.
N. Egami and E. Hartman, “Covariate Selection for Generalizing Experimental Results: Application to Large‐Scale Development Program in Uganda,” 2019, arXiv Preprint arXiv:1909.02669.
M. A. Hernán, B. C. Sauer, S. Hernández‐Díaz, R. Platt, and I. Shrier, “Specifying a Target Trial Prevents Immortal Time Bias and Other Self‐Inflicted Injuries in Observational Analyses,” Journal of Clinical Epidemiology 79 (2016): 70–75.
M. A. Hernán and J. M. Robins, Causal Inference: What if (Boca Raton, FL: Chapman & Hall/CRC, 2020).
L. E. Dang, S. Gruber, H. Lee, et al., “A Causal Roadmap for Generating High‐Quality Real‐World Evidence,” Journal of Clinical and Translational Science 7, no. 1 (2023): e212.
L. Dong, S. Yang, X. Wang, D. Zeng, and J. Cai, “Integrative Analysis of Randomized Clinical Trials With Real World Evidence Studies,” 2020, arXiv Preprint arXiv:2003.01242.
Y. Wang and J. R. Zubizarreta, “Minimal Dispersion Approximately Balancing Weights: Asymptotic Properties and Practical Considerations,” Biometrika 107, no. 1 (2020): 93–105.
E. Hartman, R. Grieve, R. Ramsahai, and J. S. Sekhon, “From Sample Average Treatment Effect to Population Average Treatment Effect on the Treated: Combining Experimental With Observational Studies to Estimate Population Treatment Effects,” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178, no. 3 (2015): 757–778.
B. Schölkopf, F. Locatello, S. Bauer, et al., “Toward Causal Representation Learning,” Proceedings of the IEEE 109, no. 5 (2021): 612–634.
T. Q. Nguyen, C. Ebnesajjad, S. R. Cole, and E. A. Stuart, “Sensitivity Analysis for an Unobserved Moderator in RCT‐To‐Target‐Population Generalization of Treatment Effects,” The Annals of Applied Statistics 1 (2017): 225–247.
T. Q. Nguyen, B. Ackerman, I. Schmid, S. R. Cole, and E. A. Stuart, “Sensitivity Analyses for Effect Modifiers Not Observed in the Target Population When Generalizing Treatment Effects From a Randomized Controlled Trial: Assumptions, Models, Effect Scales, Data Scenarios, and Implementation Details,” PLoS One 13, no. 12 (2018): e0208795.
I. J. Dahabreh, J. M. Robins, S. J. Haneuse, et al., “Sensitivity Analysis Using Bias Functions for Studies Extending Inferences From a Randomized Trial to a Target Population,” Statistics in Medicine 42 (2023): 2029–2043.
B. Colnet, J. Josse, G. Varoquaux, and E. Scornet, “Causal Effect on a Target Population: A Sensitivity Analysis to Handle Missing Covariates,” Journal of Causal Inference 10, no. 1 (2022): 372–414.
H. A. Chipman, E. I. George, R. E. McCulloch, et al., “Bart: Bayesian Additive Regression Trees,” The Annals of Applied Statistics 4, no. 1 (2010): 266–298.
T. P. Morris, I. R. White, and M. J. Crowther, “Using Simulation Studies to Evaluate Statistical Methods,” Statistics in Medicine 38, no. 11 (2019): 2074–2102.
H. Joe, Dependence Modeling With Copulas (New York: CRC Press, 2014).
J. Yan, “Enjoy the Joy of Copulas: With a Package Copula,” Journal of Statistical Software 21 (2007): 1–21.
A. Zilko, “Mixed Discrete‐Continuous Railway Disruption‐Length Models With Copulas,” 2017 (169), https://doi.org/10.4233/uuid:a551b9a2‐b5da‐4a51‐8a3b‐3d7f410d67cc.
Quasar Collaborative Group, “Adjuvant Chemotherapy Versus Observation in Patients With Colorectal Cancer: A Randomised Study,” Lancet 370, no. 9604 (2007): 2020–2029.
M. Mistry, D. M. Parkin, A. S. Ahmad, et al., “Cancer Incidence in the United Kingdom: Projections to the Year 2030,” British Journal of Cancer 105, no. 11 (2011): 1795–1803.
H. White, “A Heteroskedasticity‐Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica: Journal of the Econometric Society 48, no. 4 (1980): 817–838.
S. R. Verhoeff, F. N. van Erning, V. E. Lemmens, J. H. de Wilt, and J. F. Pruijt, “Adjuvant Chemotherapy is not Associated With Improved Survival for all High‐Risk Factors in Stage II Colon Cancer,” International Journal of Cancer 139, no. 1 (2016): 187–193.
L. Keikes, M. G. van Oijen, V. E. Lemmens, M. Koopman, and C. J. Punt, “Evaluation of Guideline Adherence in Colorectal Cancer Treatment in The Netherlands: A Survey Among Medical Oncologists by the Dutch Colorectal Cancer Group,” Clinical Colorectal Cancer 17, no. 1 (2018): 58–64.
S. Gill, C. L. Loprinzi, D. J. Sargent, et al., “Pooled Analysis of Fluorouracil‐Based Adjuvant Therapy for Stage II and III Colon Cancer: Who Benefits and by How Much?” Journal of Clinical Oncology 22, no. 10 (2004): 1797–1806.