Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions.
Confidence intervals
Misinterpretation
P-values
RCTs
Journal
BMC medical research methodology
ISSN: 1471-2288
Titre abrégé: BMC Med Res Methodol
Pays: England
ID NLM: 100968545
Informations de publication
Date de publication:
29 Oct 2024
29 Oct 2024
Historique:
received:
07
08
2024
accepted:
07
10
2024
medline:
30
10
2024
pubmed:
30
10
2024
entrez:
30
10
2024
Statut:
epublish
Résumé
Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem. We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions. We obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice. Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field. The majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences.
Sections du résumé
BACKGROUND
BACKGROUND
Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem.
METHODS
METHODS
We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions.
RESULTS
RESULTS
We obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice. Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field.
CONCLUSION
CONCLUSIONS
The majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences.
Identifiants
pubmed: 39472775
doi: 10.1186/s12874-024-02366-4
pii: 10.1186/s12874-024-02366-4
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
256Informations de copyright
© 2024. The Author(s).
Références
Adams-Huet B, Ahn C. Bridging clinical investigators and statisticians: writing the statistical methodology for a research proposal. J Investig Med. 2009;57(8):818–24. https://doi.org/10.2310/JIM.0b013e3181c2996c . PMID: 19875966; PMCID: PMC4415704.
doi: 10.2310/JIM.0b013e3181c2996c
pubmed: 19875966
pmcid: 4415704
Anderson-Cook, C.M., Lu, L. and Parker, P.A. Effective interdisciplinary collaboration between statisticians and other subject matter experts. Qual Eng. 2019;31(1):164–76.
doi: 10.1080/08982112.2018.1530357
Anney VN. Ensuring the Quality of the Findings of Qualitative Research: Looking at Trustworthiness Criteria. JETERAPS. 2014;5:272–81.
Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485.
doi: 10.1136/bmj.311.7003.485
pubmed: 7647644
pmcid: 2550545
Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305–7. https://doi.org/10.1038/d41586-019-00857-9 . (PMID: 30894741).
doi: 10.1038/d41586-019-00857-9
pubmed: 30894741
Blakeley B, McShane, Gal D, Gelman A, Robert C, Tackett J. Abandon Statistical Significance. Am Statistic. 2019;73(sup1):235–45.
doi: 10.1080/00031305.2018.1527253
Blatch-Jones A, Nuttall J, Bull A, Worswick L, Mullee M, Peveler R, Falk S, Tape N, Hinks J, Lane AJ, Wyatt JC, Griffiths G. Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK Clinical Trial Units and a qualitative study. Trials. 2020;21(1):304. https://doi.org/10.1186/s13063-020-04234-0 . PMID: 32245506; PMCID: PMC7118862.
doi: 10.1186/s13063-020-04234-0
pubmed: 32245506
pmcid: 7118862
Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058–64.
doi: 10.1001/jama.2010.651
pubmed: 20501928
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
doi: 10.1191/1478088706qp063oa
Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport, Exercise and Health. 2019;11(4):589–97. https://doi.org/10.1080/2159676X.2019.1628806 .
doi: 10.1080/2159676X.2019.1628806
Butcher NJ, Monsour A, Mew EJ, Chan AW, Moher D, Mayo-Wilson E, Terwee CB, Chee-A-Tow A, Baba A, Gavin F, Grimshaw JM, Kelly LE, Saeed L, Thabane L, Askie L, Smith M, Farid-Kapadia M, Williamson PR, Szatmari P, Tugwell P, Golub RM, Monga S, Vohra S, Marlin S, Ungar WJ, Offringa M. Guidelines for Reporting Outcomes in Trial Reports: The CONSORT-Outcomes 2022 Extension. JAMA. 2022;328(22):2252–64. https://doi.org/10.1001/jama.2022.21022 . (PMID: 36511921).
doi: 10.1001/jama.2022.21022
pubmed: 36511921
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol. 2009;114(6):1341–5. https://doi.org/10.1097/AOG.0b013e3181c3020d . (PMID: 19935040).
doi: 10.1097/AOG.0b013e3181c3020d
pubmed: 19935040
Clark T. On ‘being researched’: Why do people engage with qualitative research? Qual Res. 2010;10(4):399–419.
doi: 10.1177/1468794110366796
Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, Ashby D, Emsley R, Fergusson DA, Walters SJ, Wilson ECF, Maclennan G, Stallard N, Rothwell JC, Bland M, Brown L, Ramsay CR, Cook A, Armstrong D, Altman D, Vale LD. DELTA(2) guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. Trials. 2018;19(1):606.
doi: 10.1186/s13063-018-2884-0
pubmed: 30400926
pmcid: 6218987
Creswell JW. Controversies in mixed methods research. Sage Handbook Qual Res. 2011;4(1):269–84.
DeVito NJ, Morley J, Smith JA, Drysdale H, Goldacre B, Heneghan C. Availability of results of clinical trials registered on EU Clinical Trials Register: cross sectional audit study. BMJ Med. 2024;3(1):e000738. https://doi.org/10.1136/bmjmed-2023-000738 . (PMID:38274035;PMCID:PMC10806997).
doi: 10.1136/bmjmed-2023-000738
pubmed: 38274035
pmcid: 10806997
Dmitrienko A, Offen WW, Westfall PH. Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Stat Med. 2003;22(15):2387–400. https://doi.org/10.1002/sim.1526 . (PMID: 12872297).
doi: 10.1002/sim.1526
pubmed: 12872297
Duley L, Gillman A, Duggan M, Belson S, Knox J, McDonald A, Rawcliffe C, Simon J, Sprosen T, Watson J, Wood W. What are the main inefficiencies in trial conduct: a survey of UKCRC registered clinical trials units in the UK. Trials. 2018;19(1):15. https://doi.org/10.1186/s13063-017-2378-5 . (PMID:29310685;PMCID:PMC5759880).
doi: 10.1186/s13063-017-2378-5
pubmed: 29310685
pmcid: 5759880
Elsman EBM, Smith M, Hofstetter C, Gavin F, Jobson E, Markham S, Ricketts J, Baba A, Butcher NJ, Offringa M. A blueprint for patient and public involvement in the development of a reporting guideline for systematic reviews of outcome measurement instruments: PRISMA-COSMIN for OMIs 2024. Res Involv Engagem. 2024;10(1):33. https://doi.org/10.1186/s40900-024-00563-5 . (PMID:38515153;PMCID:PMC10956212).
doi: 10.1186/s40900-024-00563-5
pubmed: 38515153
pmcid: 10956212
Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004;6(3):e34. https://doi.org/10.2196/jmir.6.3.e34 . Erratum. In: doi: https://doi.org/10.2196/jmir.2042 . PMID:15471760;PMCID:PMC1550605.
[FDA guidance] https://www.fda.gov/media/166830/download accessed 30 Nov 2023.
Fogel DB. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review. Contemp Clin Trials Commun. 2018;1(11):156–64.
doi: 10.1016/j.conctc.2018.08.001
Gikandi A, Hallet J, Koerkamp BG, Clark CJ, Lillemoe KD, Narayan RR, Mamon HJ, Zenati MA, Wasif N, Safran DG, Besselink MG, Chang DC, Traeger LN, Weissman JS, Fong ZV. Distinguishing Clinical From Statistical Significances in Contemporary Comparative Effectiveness Research. Ann Surg. 2024;279(6):907–12. https://doi.org/10.1097/SLA.0000000000006250 . Epub 2024 Feb 23. PMID: 38390761; PMCID: PMC11087199.
doi: 10.1097/SLA.0000000000006250
pubmed: 38390761
Gardner MJ, Altman DG. Confidence intervals rather than P values:estimation rather than hypothesis testing. BMJ. 1986;292:746–50.
doi: 10.1136/bmj.292.6522.746
pubmed: 3082422
pmcid: 1339793
Goulao B, Bruhn H, Campbell M, Ramsay C, Gillies K. Patient and public involvement in numerical aspects of trials (PoINT): exploring patient and public partners experiences and identifying stakeholder priorities. Trials. 2021;22(1):499. https://doi.org/10.1186/s13063-021-05451-x . (PMID:34321066;PMCID:PMC8316879).
doi: 10.1186/s13063-021-05451-x
pubmed: 34321066
pmcid: 8316879
Gates S, Ealing E. Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals. BMJ Open. 2019;9(9):e024785.
doi: 10.1136/bmjopen-2018-024785
pubmed: 31501094
pmcid: 6738699
Gaughan M, Bozeman B. Using the prisms of gender and rank to interpret research collaboration power dynamics. Soc Stud Sci. 2016;46(4):536–58.
doi: 10.1177/0306312716652249
pubmed: 28948875
Gewandter JS, McDermott MP, Kitt RA, Chaudari J, Koch JG, Evans SR, Gross RA, Markman JD, Turk DC, Dworkin RH. Interpretation of CIs in clinical trials with non-significant results: systematic review and recommendations. BMJ Open. 2017;7(7):e017288. https://doi.org/10.1136/bmjopen-2017-017288.Review.PubMed . (PMID:28720618;PubMedCentralPMCID:PMC5726092).
doi: 10.1136/bmjopen-2017-017288.Review.PubMed
pubmed: 28720618
pmcid: 5726092
Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–50. https://doi.org/10.1007/s10654-016-0149-3 . Epub 2016 May 21. PMID: 27209009; PMCID: PMC4877414.
doi: 10.1007/s10654-016-0149-3
pubmed: 27209009
pmcid: 4877414
Hemming K, Taljaard M. Why proper understanding of confidence intervals and statistical significance is important. Med J Aust. 2021;214(3):116-118.e1.
doi: 10.5694/mja2.50926
pubmed: 33440457
Hemming K, Javid I, Taljaard M. A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common. J Clin Epidemiol. 2022;145:112–20. https://doi.org/10.1016/j.jclinepi.2022.01.014 . (Epub 2022 Jan 23 PMID: 35081450).
doi: 10.1016/j.jclinepi.2022.01.014
pubmed: 35081450
Hillen MA, Gutheil CM, Strout TD, Smets EM, Han PK. Tolerance of uncertainty: Conceptual analysis, integrative model, and implications for healthcare. Soc Sci Med. 2017;1(180):62–75.
doi: 10.1016/j.socscimed.2017.03.024
Ito C, Hashimoto A, Uemura K, Oba K. Misleading Reporting (Spin) in Noninferiority Randomized Clinical Trials in Oncology With Statistically Not Significant Results: A Systematic Review. JAMA Netw Open. 2021;4(12):e2135765. https://doi.org/10.1001/jamanetworkopen.2021.35765 . (PMID:34874407;PMCID:PMC8652604).
doi: 10.1001/jamanetworkopen.2021.35765
pubmed: 34874407
pmcid: 8652604
Kahneman D, Rosenfield AM, Gandhi L, Blaser T. Noise: How to overcome the high, hidden cost of inconsistent decision making. Harvard business review. 2016;94(10):38–46.
Langman MJS. Towards estimation and confidence intervals. BMJ. 1986;292:716.
doi: 10.1136/bmj.292.6522.716
pubmed: 3082408
pmcid: 1339775
Love SB, Yorke-Edwards V, Lensen S, Sydes MR. Monitoring in practice - How are UK academic clinical trials monitored? A survey. Trials. 2020;21(1):59. https://doi.org/10.1186/s13063-019-3976-1 . (PMID:31918743;PMCID:PMC6953230).
doi: 10.1186/s13063-019-3976-1
pubmed: 31918743
pmcid: 6953230
McDonald J, Jayasuriya R, Harris MF. The influence of power dynamics and trust on multidisciplinary collaboration: a qualitative case study of type 2 diabetes mellitus. BMC Health Serv Res. 2012;12(1):1–10.
doi: 10.1186/1472-6963-12-63
McGlothlin AE, Lewis RJ. Minimal clinically important difference: defining what really matters to patients. JAMA. 2014;312(13):1342–3.
doi: 10.1001/jama.2014.13128
pubmed: 25268441
Michie S, Van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):1–12.
doi: 10.1186/1748-5908-6-42
Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;23(340):c869. https://doi.org/10.1136/bmj.c869 . (PMID:20332511;PMCID:PMC2844943).
doi: 10.1136/bmj.c869
Monroe K, Ozyurt S, Wrigley T, Alexander A. Gender equality in academia: Bad news from the trenches, and some possible solutions. Perspect Polit. 2008;6(2):215–33.
doi: 10.1017/S1537592708080572
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research: A Synthesis of Recommendations. Acad Med. 2014;89(9):1245–51. https://doi.org/10.1097/ACM.0000000000000388 .
doi: 10.1097/ACM.0000000000000388
pubmed: 24979285
O’Cathain A, Thomas KJ. “ Any other comments?” Open questions on questionnaires–a bane or a bonus to research? BMC Med Res Methodol. 2004;4:1–7.
doi: 10.1186/1471-2288-4-25
Okpala P. Addressing power dynamics in interprofessional health care teams. Int J Healthcare Manag. 2021;14(4):1326–32.
doi: 10.1080/20479700.2020.1758894
Parker RA, Cook JA. The importance of clinical importance when determining the target difference in sample size calculations. Trials. 2023;24(1):495. https://doi.org/10.1186/s13063-023-07532-5 . (PMID:37542276;PMCID:PMC10401796).
doi: 10.1186/s13063-023-07532-5
pubmed: 37542276
pmcid: 10401796
Puhan MA, Clavien P-A. Is Statistical Significance Alone Obsolete?: Let’s Turn to Meaningful Interpretation of Scientific and Real-world Evidence on Surgical Care. Ann Surg. 2024;279(6):913–4.
pubmed: 38506046
Rawlinson C, Carron T, Cohidon C, Arditi C, Hong QN, Pluye P, Peytremann-Bridevaux I, Gilles I. An Overview of Reviews on Interprofessional Collaboration in Primary Care: Barriers and Facilitators. Int J Integr Care. 2021;21(2):32. https://doi.org/10.5334/ijic.5589 .
Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Ann Int Med. 2010;152(11):726–32.
doi: 10.7326/0003-4819-152-11-201006010-00232
pubmed: 20335313
Stone ER, Yates JF, Parker AM. Risk communication: Absolute versus relative expressions of low-probability risks. Organ Behav Hum Decis Process. 1994;60(3):387–408.
doi: 10.1006/obhd.1994.1091
Tariq S, Woodman J. Using mixed methods in health research. JRSM short reports. 2013;4(6):2042533313479197.
doi: 10.1177/2042533313479197
pubmed: 23885291
pmcid: 3697857
Vinkers CH, Lamberink HJ, Tijdink JK, Heus P, Bouter L, Glasziou P, Moher D, Damen JA, Hooft L, Otte WM. The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biol. 2021;19(4):e3001162. https://doi.org/10.1371/journal.pbio.3001162 . (PMID:33872298;PMCID:PMC8084332).
doi: 10.1371/journal.pbio.3001162
pubmed: 33872298
pmcid: 8084332
Wallerstein N, Muhammad M, Sanchez-Youngman S, Rodriguez Espinosa P, Avila M, Baker EA, Barnett S, Belone S, Golub M, Lucero J, Mahdi I. Power dynamics in community-based participatory research: A multiple–case study analysis of partnering contexts, histories, and practices. Health Educ Behav. 2019;46(1_suppl):19S-32S.
doi: 10.1177/1090198119852998
pubmed: 31549557
Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. Am Stat. 2016;70(2):129–33.
doi: 10.1080/00031305.2016.1154108
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, Tugwell P. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:1–8.
doi: 10.1186/1745-6215-13-132
Wong H. Minimum important difference is minimally important in sample size calculations. Trials. 2023;24:34. https://doi.org/10.1186/s13063-023-07092-8 .
doi: 10.1186/s13063-023-07092-8
pubmed: 36650587
pmcid: 9847050
Young PJ, Nickson CP, Perner A. When Should Clinicians Act on Non-Statistically Significant Results From Clinical Trials? [published online ahead of print, 2020 May 8]. JAMA. 2020; https://doi.org/10.1001/jama.2020.3508 .