A Bayesian analysis of non-significant rehabilitation findings: Evaluating the evidence in favour of truly absent treatment effects.
Bayes factor
Bayesian analysis
Meta-analysis
Meta-research
Null hypothesis significance testing
Statistical power
Journal
Annals of physical and rehabilitation medicine
ISSN: 1877-0665
Titre abrégé: Ann Phys Rehabil Med
Pays: Netherlands
ID NLM: 101502773
Informations de publication
Date de publication:
Jul 2021
Jul 2021
Historique:
received:
28
03
2020
revised:
06
05
2020
accepted:
12
07
2020
pubmed:
18
8
2020
medline:
21
10
2021
entrez:
18
8
2020
Statut:
ppublish
Résumé
Relying solely on null hypothesis significance testing (NHST) to investigate rehabilitation interventions may result in researchers erroneously concluding the absence of a treatment effect. We aimed to distinguish between truly null treatment effects and data that are insensitive to detecting treatment effects by calculating Bayes factors (BF We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that clearly evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups for individual findings within 175 meta-analyses. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. We calculated t-statistics and associated P-values for each finding in order to extract non-significant results (P>0.05). We calculated BF Across all intervention types, most (71.9%) findings were deemed anecdotal, and this pattern remained within distinct intervention types (58.4-76.0%). Larger sample sizes tended to be associated with greater strength in favour of the null hypothesis, both across and within intervention types. Larger P-values were not associated with greater strength in favour of the null hypothesis; this finding was present both across and within intervention types. Our findings indicate that most non-significant rehabilitation findings are unable to distinguish between the true absence of a treatment effect and data that are merely insensitive to detecting a treatment effect. Findings also suggest that rehabilitation researchers may improve the strength of their statistical conclusions by increasing sample size and that Bayes factors may offer unique benefits relative to P-values.
Sections du résumé
BACKGROUND
BACKGROUND
Relying solely on null hypothesis significance testing (NHST) to investigate rehabilitation interventions may result in researchers erroneously concluding the absence of a treatment effect.
OBJECTIVE
OBJECTIVE
We aimed to distinguish between truly null treatment effects and data that are insensitive to detecting treatment effects by calculating Bayes factors (BF
METHOD
METHODS
We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that clearly evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups for individual findings within 175 meta-analyses. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. We calculated t-statistics and associated P-values for each finding in order to extract non-significant results (P>0.05). We calculated BF
RESULTS
RESULTS
Across all intervention types, most (71.9%) findings were deemed anecdotal, and this pattern remained within distinct intervention types (58.4-76.0%). Larger sample sizes tended to be associated with greater strength in favour of the null hypothesis, both across and within intervention types. Larger P-values were not associated with greater strength in favour of the null hypothesis; this finding was present both across and within intervention types.
CONCLUSION
CONCLUSIONS
Our findings indicate that most non-significant rehabilitation findings are unable to distinguish between the true absence of a treatment effect and data that are merely insensitive to detecting a treatment effect. Findings also suggest that rehabilitation researchers may improve the strength of their statistical conclusions by increasing sample size and that Bayes factors may offer unique benefits relative to P-values.
Identifiants
pubmed: 32805456
pii: S1877-0657(20)30155-X
doi: 10.1016/j.rehab.2020.07.008
pii:
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
101425Informations de copyright
Copyright © 2020. Published by Elsevier Masson SAS.