'Not clinically effective but cost-effective' - paradoxical conclusions in randomised controlled trials with 'doubly null' results: a cross-sectional study.

Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: 'the intervention is not clinically effective' but 'is probably cost-effective'. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them. Cross-sectional retrospective analysis of randomised trials published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as 'doubly null' those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online. The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one 'doubly null' comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9). Trials with 'doubly null' results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed. Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with 'doubly null' results, particularly when economics favours a novel intervention.

Informations de copyright

© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Déclaration de conflit d'intérêts

Competing interests: All authors work part-time for the NIHR HTA programme as well as being independent academics. All but HW and AC are current members of the editorial board of the HTA monograph series. HW is director of the HTA programme. AC is a past member of the editorial board of the HTA monograph series. JR is employed part-time as a researcher by NETSCC, the secretariat which manages the NIHR HTA programme. Neither NIHR nor the HTA programme had any role in interpreting the results nor in preparing this paper for publication. NETSCC has taken responsibility for payment of the publication fee.