Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study.
efficacy
meta-epidemiological study
pilot trial
power calculation
randomized controlled trial
sample size calculation
Journal
BMJ evidence-based medicine
ISSN: 2515-4478
Titre abrégé: BMJ Evid Based Med
Pays: England
ID NLM: 101719009
Informations de publication
Date de publication:
22 11 2023
22 11 2023
Historique:
accepted:
09
07
2023
medline:
24
11
2023
pubmed:
26
7
2023
entrez:
25
7
2023
Statut:
epublish
Résumé
Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial. We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial. A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004). Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.
Sections du résumé
BACKGROUND
Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial.
METHODS
We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial.
RESULTS
A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004).
CONCLUSION
Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.
Identifiants
pubmed: 37491141
pii: bmjebm-2023-112358
doi: 10.1136/bmjebm-2023-112358
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
383-391Informations de copyright
© Author(s) (or their employer(s)) 2023. No commercial re-use. See rights and permissions. Published by BMJ.
Déclaration de conflit d'intérêts
Competing interests: None declared.