How to specify healthcare process improvements collaboratively using rapid, remote consensus-building: a framework and a case study of its application.
Best practices
COVID-19
Consensus development
Consensus-building
Delphi technique
Obstetrics
Postpartum haemorrhage
Professional practice
Journal
BMC medical research methodology
ISSN: 1471-2288
Titre abrégé: BMC Med Res Methodol
Pays: England
ID NLM: 100968545
Informations de publication
Date de publication:
11 05 2021
11 05 2021
Historique:
received:
06
01
2021
accepted:
21
04
2021
entrez:
12
5
2021
pubmed:
13
5
2021
medline:
25
6
2021
Statut:
epublish
Résumé
Practical methods for facilitating process improvement are needed to support high quality, safe care. How best to specify (identify and define) process improvements - the changes that need to be made in a healthcare process - remains a key question. Methods for doing so collaboratively, rapidly and remotely offer much potential, but are under-developed. We propose an approach for engaging diverse stakeholders remotely in a consensus-building exercise to help specify improvements in a healthcare process, and we illustrate the approach in a case study. Organised in a five-step framework, our proposed approach is informed by a participatory ethos, crowdsourcing and consensus-building methods: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvement in light of learning from the exercise. We tested the approach in a case study that sought to specify process improvements for the management of obstetric emergencies during the COVID-19 pandemic. We used a brief video showing a process for managing a post-partum haemorrhage in women with COVID-19 to elicit recommendations on how the process could be improved. Two Delphi rounds were then conducted to reach consensus. We gathered views from 105 participants, with a background in maternity care (n = 36), infection prevention and control (n = 17), or human factors (n = 52). The participants initially generated 818 recommendations for how to improve the process illustrated in the video, which we synthesised into a set of 22 recommendations. The consensus-building exercise yielded a final set of 16 recommendations. These were used to inform the specification of process improvements for managing the obstetric emergency and develop supporting resources, including an updated video. The proposed methodological approach enabled the expertise and ingenuity of diverse stakeholders to be captured and mobilised to specify process improvements in an area of pressing service need. This approach has the potential to address current challenges in process improvement, but will require further evaluation.
Sections du résumé
BACKGROUND
Practical methods for facilitating process improvement are needed to support high quality, safe care. How best to specify (identify and define) process improvements - the changes that need to be made in a healthcare process - remains a key question. Methods for doing so collaboratively, rapidly and remotely offer much potential, but are under-developed. We propose an approach for engaging diverse stakeholders remotely in a consensus-building exercise to help specify improvements in a healthcare process, and we illustrate the approach in a case study.
METHODS
Organised in a five-step framework, our proposed approach is informed by a participatory ethos, crowdsourcing and consensus-building methods: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvement in light of learning from the exercise. We tested the approach in a case study that sought to specify process improvements for the management of obstetric emergencies during the COVID-19 pandemic. We used a brief video showing a process for managing a post-partum haemorrhage in women with COVID-19 to elicit recommendations on how the process could be improved. Two Delphi rounds were then conducted to reach consensus.
RESULTS
We gathered views from 105 participants, with a background in maternity care (n = 36), infection prevention and control (n = 17), or human factors (n = 52). The participants initially generated 818 recommendations for how to improve the process illustrated in the video, which we synthesised into a set of 22 recommendations. The consensus-building exercise yielded a final set of 16 recommendations. These were used to inform the specification of process improvements for managing the obstetric emergency and develop supporting resources, including an updated video.
CONCLUSIONS
The proposed methodological approach enabled the expertise and ingenuity of diverse stakeholders to be captured and mobilised to specify process improvements in an area of pressing service need. This approach has the potential to address current challenges in process improvement, but will require further evaluation.
Identifiants
pubmed: 33975550
doi: 10.1186/s12874-021-01288-9
pii: 10.1186/s12874-021-01288-9
pmc: PMC8111055
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
103Subventions
Organisme : Health Foundation
ID : This project was supported by the Health Foundation's grant to the University of Cambridge for The Healthcare Improvement Studies (THIS) Institute.
Investigateurs
André Sartori
(A)
Andy Paterson
(A)
Doro Unger-Lee
(D)
Joann Leeding
(J)
Luke Steer
(L)
Amanda Andrews
(A)
Rita Arya
(R)
Sarah F Bell
(SF)
Denise Chaffer
(D)
Andrew Cooney
(A)
Rachel Corry
(R)
Mair G P Davies
(MGP)
Lisa Duffy
(L)
Caroline Everden
(C)
Theresa Fitzpatrick
(T)
Courtney Grant
(C)
Mark Hellaby
(M)
Tracey A Herlihey
(TA)
Sue Hignett
(S)
Sarah Hookes
(S)
Fran R Ives
(FR)
Gyuchan T Jun
(GT)
Owen J Marsh
(OJ)
Tanya R Matthews
(TR)
Celine McKeown
(C)
Alexandra Merriman
(A)
Giulia Miles
(G)
Susan Millward
(S)
Neil Muchatata
(N)
David Newton
(D)
Valerie G Noble
(VG)
Pamela Page
(P)
Vincent Pargade
(V)
Sharon P Pickering
(SP)
Laura Pickup
(L)
Dale Richards
(D)
Cerys Scarr
(C)
Jyoti Sidhu
(J)
James Stevenson
(J)
Ben Tipney
(B)
Stephen Tipper
(S)
Jo Wailling
(J)
Susan P Whalley-Lloyd
(SP)
Christian Wilhelm
(C)
Juliet J Wood
(JJ)
Commentaires et corrections
Type : ErratumIn
Références
Brouwers MC, Spithoff K, Kerkvliet K, Alonso-Coello P, Burgers J, Cluzeau F, et al. Development and validation of a tool to assess the quality of clinical practice guideline recommendations. JAMA Netw Open. 2020;3(5):e205535.
doi: 10.1001/jamanetworkopen.2020.5535
Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8. https://doi.org/10.1093/intqhc/mzv115 .
doi: 10.1093/intqhc/mzv115
pubmed: 4767049
pmcid: 4767049
Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4(1):212. https://doi.org/10.1186/1756-0500-4-212 .
doi: 10.1186/1756-0500-4-212
pubmed: 3148986
pmcid: 3148986
Grol R, Wensing M, Grimshaw J. Improving patient care: the implementation of change in health care (third edition). Oxford: Wiley; 2020.
Chassin MR, Loeb JM. High-reliability health care: getting there from Here. Milbank Q. 2013;91(3):459–90. https://doi.org/10.1111/1468-0009.12023 .
doi: 10.1111/1468-0009.12023
pubmed: 3790522
pmcid: 3790522
Holden RJ, Carayon P, Gurses AP, Hoonakker P, Hundt AS, Ozok AA, et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics. 2013;56(11):1669–86. https://doi.org/10.1080/00140139.2013.838643 .
doi: 10.1080/00140139.2013.838643
Alfred M, Catchpole K, Huffer E, Fredendall L, Taaffe KM. Work systems analysis of sterile processing: decontamination. BMJ Qual Saf. 2020;29(4):320–8. https://doi.org/10.1136/bmjqs-2019-009422 .
doi: 10.1136/bmjqs-2019-009422
Antonacci G, Reed JE, Lennox L, Barlow J. The use of process mapping in healthcare quality improvement projects. Health Serv Manag Res. 2018;31(2):74–84. https://doi.org/10.1177/0951484818770411 .
doi: 10.1177/0951484818770411
The Health Foundation. Quality improvement made simple: What everyone should know about health care quality improvement. 2013. Available at: https://www.health.org.uk/publications/quality-improvement-made-simple .
Boaden R, Harvey G, Moxham C, Proudlove N: Quality improvement: theory and practice in healthcare. 2008. Available at: https://www.england.nhs.uk/improvement-hub/publication/quality-improvement-theory-practice-in-healthcare/ .
Hughes RG, Ronda G. Tools and strategies for quality improvement and patient safety. In: Patient safety and quality: An evidence-based handbook for nurses. US: Agency for Healthcare Research and Quality; 2008.
Johnson JK, Barach P. Tools and strategies for continuous quality improvement and patient safety. In: Surgical Patient Care: Springer; 2017. p. 121–32.
Clegg CW. Sociotechnical principles for system design. Appl Ergon. 2000;31(5):463–77. https://doi.org/10.1016/S0003-6870(00)00009-0 .
doi: 10.1016/S0003-6870(00)00009-0
Karsh BT, Waterson P, Holden RJ. Crossing levels in systems ergonomics: a framework to support 'mesoergonomic' inquiry. Appl Ergon. 2014;45(1):45–54. https://doi.org/10.1016/j.apergo.2013.04.021 .
doi: 10.1016/j.apergo.2013.04.021
Royal Academy of Engineering. Engineering better care: A systems approach to health and care design and continuous improvement. 2017. Available at: https://www.raeng.org.uk/publications/reports/engineering-better-care .
Dixon-Woods M, Pronovost PJ. Patient safety and the problem of many hands. BMJ Qual Saf. 2016;25(7):485–8. https://doi.org/10.1136/bmjqs-2016-005232 .
doi: 10.1136/bmjqs-2016-005232
pubmed: 4959572
pmcid: 4959572
Dixon-Woods M. Harveian oration 2018: improving quality and safety in healthcare. Clin Med. 2019;19(1):47–56. https://doi.org/10.7861/clinmedicine.19-1-47 .
doi: 10.7861/clinmedicine.19-1-47
Dixon-Woods M, Martin GP. Does quality improvement improve quality? Future Hosp J. 2016;3(3):191–4. https://doi.org/10.7861/futurehosp.3-3-191 .
doi: 10.7861/futurehosp.3-3-191
pubmed: 6465806
pmcid: 6465806
Shorrock S. The loneliest profession in healthcare. 2018. Available at: https://humanisticsystems.com/2018/05/11/the-loneliest-profession-in-healthcare/ .
Gurses AP, Tschudy MM, McGrath-Morrow S, Husain A, Solomon BS, Gerohristodoulos KA, et al. Overcoming COVID-19: what can human factors and ergonomics offer? J Patient Saf Risk Manage. 2020;25(2):49–54. https://doi.org/10.1177/2516043520917764 .
doi: 10.1177/2516043520917764
Hignett S, Jones EL, Miller D, Wolf L, Modi C, Shahzad MW, et al. Human factors and ergonomics and quality improvement science: integrating approaches for safety in healthcare. BMJ Qual Saf. 2015;24(4):250–4. https://doi.org/10.1136/bmjqs-2014-003623 .
doi: 10.1136/bmjqs-2014-003623
pubmed: 25715799
pmcid: 25715799
Mukamel DB, Haeder SF, Weimer DL. Top-down and bottom-up approaches to health care quality: the impacts of regulation and report cards. Annu Rev Public Health. 2014;35(1):477–97. https://doi.org/10.1146/annurev-publhealth-082313-115826 .
doi: 10.1146/annurev-publhealth-082313-115826
pubmed: 24159921
pmcid: 24159921
Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–33. https://doi.org/10.1056/NEJMoa2001017 .
doi: 10.1056/NEJMoa2001017
pubmed: 31978945
pmcid: 31978945
World Health Organization. Timeline: WHO’s COVID-19 response. 2020. Available at: https://www.who.int/news-room/detail/29-06-2020-covidtimeline .
Indini A, Aschele C, Cavanna L, Clerico M, Daniele B, Fiorentini G, et al. Reorganisation of medical oncology departments during the novel coronavirus disease-19 pandemic: a nationwide Italian survey. Eur J Cancer. 2020;132:17–23. https://doi.org/10.1016/j.ejca.2020.03.024 .
doi: 10.1016/j.ejca.2020.03.024
pubmed: 7136865
pmcid: 7136865
Hanna TP, Evans GA, Booth CM. Cancer, COVID-19 and the precautionary principle: prioritizing treatment during a global pandemic. Nat Rev Clin Oncol. 2020;17(5):268–70. https://doi.org/10.1038/s41571-020-0362-6 .
doi: 10.1038/s41571-020-0362-6
Royal College of Surgeons of England. Recovery of surgical services during and after COVID-19. 2020. Available at: https://www.rcseng.ac.uk/coronavirus/recovery-of-surgical-services/ .
Coimbra R, Edwards S, Coimbra BC, Tabuenca A. Resuming elective surgical services in times of COVID-19 infection. Trauma Surg Acute Care Open. 2020;5(1):e000511. https://doi.org/10.1136/tsaco-2020-000511 .
doi: 10.1136/tsaco-2020-000511
American College of Surgeons, American Society of Anesthesiologists, Association of periOperative Registered Nurses, American Hospital Association. Joint statement: roadmap for resuming elective surgery after COVID-19 pandemic. 2020. Available at: https://www.facs.org/covid-19/clinical-guidance/roadmap-elective-surgery .
Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiatry. 2020;7(9):813–24. https://doi.org/10.1016/S2215-0366(20)30307-2 .
doi: 10.1016/S2215-0366(20)30307-2
pubmed: 32682460
pmcid: 32682460
Royal College of Midwives, Royal College of Obstetricians & Gynaecologists. Guidance for provision of midwife-led settings and home birth in the evolving coronavirus (COVID-19) pandemic. 2020. Available at: https://www.evidence.nhs.uk/document?id=2273544&returnUrl=search%3Fq%3Dhomebirth .
NHS England. Clinical guide for the temporary reorganisation of intrapartum maternity care during the coronavirus pandemic. 2020. Available at: https://madeinheene.hee.nhs.uk/Portals/0/Clinical%20guide%20for%20the%20temporary%20reorganisation%20of%20intrapartum%20maternity%20care.pdf .
Semaan A, Audet C, Huysmans E, Afolabi B, Assarag B, Banke-Thomas A, et al. Voices from the frontline: findings from a thematic analysis of a rapid online global survey of maternal and newborn health professionals facing the COVID-19 pandemic. BMJ Glob Health. 2020;5(6):e002967. https://doi.org/10.1136/bmjgh-2020-002967 .
doi: 10.1136/bmjgh-2020-002967
Centers for Disease Control and Prevention (CDC). Considerations for inpatient obstetric healthcare settings. 2020. Available at: https://www.cdc.gov/coronavirus/2019-ncov/hcp/inpatient-obstetric-healthcare-guidance.html .
Chen D, Yang H, Cao Y, Cheng W, Duan T, Fan C, et al. Expert consensus for managing pregnant women and neonates born to mothers with suspected or confirmed novel coronavirus ( COVID −19) infection. Int J Gynecol Obstet. 2020;149(2):130–6. https://doi.org/10.1002/ijgo.13146 .
doi: 10.1002/ijgo.13146
Dixon-Woods M. How to improve healthcare improvement—an essay by Mary Dixon-Woods. BMJ. 2019;367:l5514.
doi: 10.1136/bmj.l5514
Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801. https://doi.org/10.1111/hex.12888 .
doi: 10.1111/hex.12888
pubmed: 31012259
pmcid: 31012259
Aveling EL, Martin G, Armstrong N, Banerjee J, Dixon-Woods M. Quality improvement through clinical communities: eight lessons for practice. J Health Organ Manage. 2012;26(2):158–74. https://doi.org/10.1108/14777261211230754 .
doi: 10.1108/14777261211230754
Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89(2):167–205. https://doi.org/10.1111/j.1468-0009.2011.00625.x .
doi: 10.1111/j.1468-0009.2011.00625.x
pubmed: 21676020
pmcid: 21676020
Fowler A. Authentic NGDO partnerships in the new policy agenda for international aid: dead end or light ahead? Dev Chang. 1998;29(1):137–59. https://doi.org/10.1111/1467-7660.00073 .
doi: 10.1111/1467-7660.00073
Kahssay HM, Oakley P, World Health Organization. Community involvement in health development: a review of the concept and practice. 1999. Available at: https://apps.who.int/iris/handle/10665/42151 .
Marjanovic S, Ball S, Harshfield A, Dimova S, Prideaux R, Carpenter A, Punch D, Simmons RK. Involving NHS staff in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/involving-nhs-staff-in-research/ .
Eyre L, Farrelly M, Marshall M. What can a participatory approach to evaluation contribute to the field of integrated care? BMJ Qual Saf. 2017;26(7):588–94. https://doi.org/10.1136/bmjqs-2016-005777 .
doi: 10.1136/bmjqs-2016-005777
Parks S, d'Angelo C, Gunashekar S. Citizen science: generating ideas and exploring consensus. The Healthcare Improvement Studies (THIS) Institute 2018. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/citizen-science-generating-ideas-exploring-consensus/
Guazzini A, Vilone D, Donati C, Nardi A, Levnajić Z. Modeling crowdsourcing as collective problem solving. Sci Rep. 2015;5(1):–16557. https://doi.org/10.1038/srep16557 .
Franzoni C, Sauermann H. Crowd science: the organization of scientific research in open collaborative projects. Res Policy. 2014;43(1):1–20. https://doi.org/10.1016/j.respol.2013.07.005 .
doi: 10.1016/j.respol.2013.07.005
Breidbach CF, Brodie RJ. Engagement platforms in the sharing economy. J Serv Theory Pract. 2017;27(4):761–77. https://doi.org/10.1108/JSTP-04-2016-0071 .
doi: 10.1108/JSTP-04-2016-0071
Cappa F, Laut J, Porfiri M, Giustiniano L. Bring them aboard: rewarding participation in technology-mediated citizen science projects. Comput Hum Behav. 2018;89:246–57. https://doi.org/10.1016/j.chb.2018.08.017 .
doi: 10.1016/j.chb.2018.08.017
European Citizen Science Association. Ten principles of citizen science. 2015. Available at: https://osf.io/xpr2n/ .
Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018. https://doi.org/10.2307/j.ctv550cf2 .
doi: 10.2307/j.ctv550cf2
Lichten C, Ioppolo R, D'Angelo C, Simmons RK, Jones MM. Citizen science: crowdsourcing for research. The Healthcare Improvement Studies (THIS) Institute 2018. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/citizen-science-crowdsourcing-research/ .
Jones J, Hunter D. Qualitative research: consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80. https://doi.org/10.1136/bmj.311.7001.376 .
doi: 10.1136/bmj.311.7001.376
pubmed: 7640549
pmcid: 7640549
Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2008;26(1):5–14. https://doi.org/10.1111/j.1365-2710.2001.00331.x .
doi: 10.1111/j.1365-2710.2001.00331.x
Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(S3):280.
doi: 10.1186/s13063-017-1978-4
Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):1–88.
doi: 10.3310/hta2030
Innes JE. Planning through consensus building: a new view of the comprehensive planning ideal. J Am Plan Assoc. 1996;62(4):460–72. https://doi.org/10.1080/01944369608975712 .
doi: 10.1080/01944369608975712
Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94. https://doi.org/10.1111/j.1440-172X.2012.02017.x .
doi: 10.1111/j.1440-172X.2012.02017.x
pubmed: 22435983
pmcid: 22435983
Van De Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–42. https://doi.org/10.2105/ajph.62.3.337 .
doi: 10.2105/ajph.62.3.337
pubmed: 5011164
pmcid: 5011164
Lomas J, Anderson G, Enkin M, Vayda E, Roberts R, MacKinnon B. The role of evidence in the consensus process: results from a Canadian consensus exercise. JAMA. 1988;259(20):3001–5. https://doi.org/10.1001/jama.1988.03720200023027 .
doi: 10.1001/jama.1988.03720200023027
Institute of Medicine (US) Council on Health Care Technology, Goodman C, Baratz SR. Improving consensus development for health technology assessment: An international perspective. Washington: National Academies Press (US); 1990.
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. The RAND/UCLA appropriateness method user's manual. Santa Monica: Rand Corp; 2001.
Brook RH. Assessing the appropriateness of care—its time has come. JAMA. 2009;302(9):997–8. https://doi.org/10.1001/jama.2009.1279 .
doi: 10.1001/jama.2009.1279
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67. https://doi.org/10.1287/mnsc.9.3.458 .
doi: 10.1287/mnsc.9.3.458
Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang. 2006;73(5):467–82. https://doi.org/10.1016/j.techfore.2005.09.002 .
doi: 10.1016/j.techfore.2005.09.002
Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48. https://doi.org/10.1177/135581969900400410 .
doi: 10.1177/135581969900400410
pubmed: 10623041
pmcid: 10623041
Banno M, Tsujimoto Y, Kataoka Y. The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines. J Clin Epidemiol. 2020;124:50–7. https://doi.org/10.1016/j.jclinepi.2020.04.010 .
doi: 10.1016/j.jclinepi.2020.04.010
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of Health Research reporting guidelines. PLoS Med. 2010;7(2):e1000217. https://doi.org/10.1371/journal.pmed.1000217 .
doi: 10.1371/journal.pmed.1000217
pubmed: 2821895
pmcid: 2821895
Hsu C-C, Sandford BA. The Delphi Technique: Making Sense of Consensus. Pract Assess Res Eval. 2007;12:10.
Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12. https://doi.org/10.1111/j.1365-2648.2006.03716.x .
doi: 10.1111/j.1365-2648.2006.03716.x
pubmed: 16422719
pmcid: 16422719
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. https://doi.org/10.1371/journal.pone.0020476 .
doi: 10.1371/journal.pone.0020476
pubmed: 3111406
pmcid: 3111406
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.
Brouwers MC, Kerkvliet K, Spithoff K. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016:i1152. https://doi.org/10.1136/bmj.i1152 .
Graham ID, Kothari A, McCutcheon C. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13(1):1.
doi: 10.1186/s13012-017-0700-y
Brereton L, Ingleton C, Gardiner C, Goyder E, Mozygemba K, Lysdahl KB, et al. Lay and professional stakeholder involvement in scoping palliative care issues: methods used in seven European countries. Palliat Med. 2017;31(2):181–92. https://doi.org/10.1177/0269216316649154 .
doi: 10.1177/0269216316649154
Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. https://doi.org/10.1177/0269216317690685 .
doi: 10.1177/0269216317690685
Herodotou C, Aristeidou M, Sharples M, Scanlon E. Designing citizen science tools for learning: lessons learnt from the iterative development of nQuire. Res Pract Technol Enhanc Learn. 2018;13(1):1.
doi: 10.1186/s41039-018-0072-1
Wald DM, Longo J, Dobell AR. Design principles for engaging and retaining virtual citizen scientists. Conserv Biol. 2016;30(3):562–70. https://doi.org/10.1111/cobi.12627 .
doi: 10.1111/cobi.12627
pubmed: 26365126
pmcid: 26365126
Boers M, Kirwan JR, Wells G, Beaton D, Gossec L, D'Agostino M-A, et al. Developing Core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67(7):745–53. https://doi.org/10.1016/j.jclinepi.2013.11.013 .
doi: 10.1016/j.jclinepi.2013.11.013
pubmed: 24582946
pmcid: 24582946
Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x .
doi: 10.1111/j.1365-2648.1987.tb01376.x
pubmed: 3320139
pmcid: 3320139
Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82. https://doi.org/10.1046/j.1365-2648.2003.02537.x .
doi: 10.1046/j.1365-2648.2003.02537.x
Hussler C, Muller P, Rondé P. Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technol Forecast Soc Chang. 2011;78(9):1642–53. https://doi.org/10.1016/j.techfore.2011.07.008 .
doi: 10.1016/j.techfore.2011.07.008
Ioannidis JPA. Professional societies should abstain from authorship of guidelines and disease definition statements. Circulation. 2018;11(10):e004889.
Hutchings A, Raine R. A systematic review of factors affecting the judgments produced by formal consensus development methods in health care. J Health Serv Res Policy. 2006;11(3):172–179H. https://doi.org/10.1258/135581906777641659 .
doi: 10.1258/135581906777641659
Marjanovic S, Harshfield A, Carpenter A, Bertscher A, Punch D, Ball S. Involving patients and the public in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/involving-patients-public-research/ .
Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5(1):37.
doi: 10.1186/1471-2288-5-37
Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7(1):52. https://doi.org/10.1186/1471-2288-7-52 .
doi: 10.1186/1471-2288-7-52
pubmed: 2216026
pmcid: 2216026
West S, Pateman R. Recruiting and retaining participants in citizen science: what can be learned from the volunteering literature? Citiz Sci. 2016;1(2):15.
Brouwer S, Hessels LK. Increasing research impact with citizen science: the influence of recruitment strategies on sample diversity. Public Underst Sci. 2019;28(5):606–21. https://doi.org/10.1177/0963662519840934 .
doi: 10.1177/0963662519840934
Fiske A, Prainsack B, Buyx A. Meeting the needs of underserved populations: setting the agenda for more inclusive citizen science of medicine. J Med Ethics. 2019;45(9):617–22. https://doi.org/10.1136/medethics-2018-105253 .
doi: 10.1136/medethics-2018-105253
De Las ND, Hacker K, DiGirolamo A, Hicks LS. A systematic review of community-based participatory research to enhance clinical trials in racial and ethnic minority groups. Health Serv Res. 2012;47(3pt2):1363–86.
doi: 10.1111/j.1475-6773.2012.01386.x
Hsu C-C, Sandford BA. Minimizing non-response in the Delphi process: How to respond to non-response. Pract Assess Res Eval. 2007;12(1):17.
Ogrinc G, Nelson WA, Adams SM, O'Hara AE. An instrument to differentiate between clinical research and quality improvement. Irb. 2013;35(5):1–8.
Healthcare Quality Improvement Partnership (HQIP). Guide to managing ethical issues in quality improvement or clinical audit projects. 2021. Available at: https://www.hqip.org.uk/resource/guide-to-managing-ethical-issues-in-quality-improvement-or-clinical-audit-projects/#.YGTi769KhnI .
Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decis Sci. 1997;28(3):763–74. https://doi.org/10.1111/j.1540-5915.1997.tb01330.x .
doi: 10.1111/j.1540-5915.1997.tb01330.x
Schmidt R, Lyytinen K, Keil M, Cule P. Identifying software project risks: an international Delphi study. J Manag Inf Syst. 2001;17(4):5–36. https://doi.org/10.1080/07421222.2001.11045662 .
doi: 10.1080/07421222.2001.11045662
Brady SR. Utilizing and adapting the Delphi method for use in qualitative research. Int J Qual Methods. 2015;14(5):160940691562138. https://doi.org/10.1177/1609406915621381 .
doi: 10.1177/1609406915621381
Hirschhorn F. Reflections on the application of the Delphi method: lessons from a case in public transport research. Int J Soc Res Methodol. 2019;22(3):309–22. https://doi.org/10.1080/13645579.2018.1543841 .
doi: 10.1080/13645579.2018.1543841
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9. https://doi.org/10.1016/j.jclinepi.2013.12.002 .
doi: 10.1016/j.jclinepi.2013.12.002
pubmed: 24581294
Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19(1):173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173 .
doi: 10.1146/annurev.publhealth.19.1.173
Renfrew MJ, Cheyne H, Craig J, Duff E, Dykes F, Hunter B, et al. Sustaining quality midwifery care in a pandemic and beyond. Midwifery. 2020;88:102759. https://doi.org/10.1016/j.midw.2020.102759 .
doi: 10.1016/j.midw.2020.102759
pubmed: 32485502
pmcid: 32485502
Coxon K, Turienzo CF, Kweekel L, Goodarzi B, Brigante L, Simon A, et al. The impact of the coronavirus (COVID-19) pandemic on maternity care in Europe. Midwifery. 2020;88:102779.
doi: 10.1016/j.midw.2020.102779
Knight M, Bunch K, Vousden N, Morris E, Simpson N, Gale C, et al. Characteristics and outcomes of pregnant women admitted to hospital with confirmed SARS-CoV-2 infection in UK: national population based cohort study. BMJ. 2020:m2107. https://doi.org/10.1136/bmj.m2107 .
Royal College of Obstetricians & Gynaecologists, Royal College of Midwives, Royal College of Paediatrics and Child Health, Public Health England, Public Health Scotland. Coronavirus (COVID-19) infection in pregnancy. 2020. Available at: https://www.rcog.org.uk/coronavirus-pregnancy .
Rimmer M, Al Wattar B, Barlow C, Black N, Carpenter C, Conti-Ramsden F, et al. Provision of obstetrics and gynaecology services during the COVID-19 pandemic: a survey of junior doctors in the UK National Health Service. BJOG Int J Obstet Gynaecol. 2020;127(9):1123–8. https://doi.org/10.1111/1471-0528.16313 .
doi: 10.1111/1471-0528.16313
Mehrabadi A, Hutcheon J, Lee L, Kramer M, Liston R, Joseph K. Epidemiological investigation of a temporal increase in atonic postpartum haemorrhage: a population-based retrospective cohort study. BJOG Int J Obstet Gynaecol. 2013;120(7):853–62. https://doi.org/10.1111/1471-0528.12149 .
doi: 10.1111/1471-0528.12149
van Stralen G, von Schmidt auf Altenstadt JF, Bloemenkamp KW, van Roosmalen J, Hukkelhoven CW. Increasing incidence of postpartum hemorrhage: the Dutch piece of the puzzle. Acta Obstet Gynecol Scand. 2016;95(10):1104–10. https://doi.org/10.1111/aogs.12950 .
doi: 10.1111/aogs.12950
pubmed: 27460955
pmcid: 27460955
Knight M, Bunch K, Tuffnell D, Jayakody H, Shakespeare J, Kotnis R, et al. Saving Lives, Improving Mothers’ Care - Lessons learned to inform maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2014–16. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2018.
Dunning T, Harris JM, Sandall J. Women and their birth partners’ experiences following a primary postpartum haemorrhage: a qualitative study. BMC Pregnancy Childbirth. 2016;16(1):80.
doi: 10.1186/s12884-016-0870-7
Say L, Chou D, Gemmill A, Tunçalp Ö, Moller A-B, Daniels J, et al. Global causes of maternal death: a WHO systematic analysis. Lancet Glob Health. 2014;2(6):e323–33. https://doi.org/10.1016/S2214-109X(14)70227-X .
doi: 10.1016/S2214-109X(14)70227-X
pubmed: 25103301
pmcid: 25103301
Zwart JJ, Dupuis JRO, Richters A, Öry F, Van Roosmalen J. Obstetric intensive care unit admission: a 2-year nationwide population-based cohort study. Intensive Care Med. 2010;36(2):256–63. https://doi.org/10.1007/s00134-009-1707-x .
doi: 10.1007/s00134-009-1707-x
Mavrides E, Allard S, Chandraharan E, Collins P, Green L, Hunt B, et al. Prevention and management of postpartum Haemorrhage. BJOG Int J Obstet Gynaecol. 2016;124(5):e106–49.
National Institute for Health and Care Excellence (NICE). Intrapartum care for healthy women and babies. 2017. Available at: https://www.nice.org.uk/guidance/CG190 .
Gayet-Ageron A, Prieto-Merino D, Ker K, Shakur H, Ageron F-X, Roberts I, et al. Effect of trekeratment delay on the effectiveness and safety of antifibrinolytics in acute severe haemorrhage: a meta-analysis of individual patient-level data from 40 138 bleeding patients. Lancet. 2018;391(10116):125–32. https://doi.org/10.1016/S0140-6736(17)32455-8 .
doi: 10.1016/S0140-6736(17)32455-8
pubmed: 5773762
pmcid: 5773762
Shakur H, Roberts I, Fawole B, Chaudhri R, El-Sheikh M, Akintan A, et al. Effect of early tranexamic acid administration on mortality, hysterectomy, and other morbidities in women with post-partum haemorrhage (WOMAN): an international, randomised, double-blind, placebo-controlled trial. Lancet. 2017;389(10084):2105–16. https://doi.org/10.1016/S0140-6736(17)30638-4 .
doi: 10.1016/S0140-6736(17)30638-4
Liberati EG, Tarrant C, Willars J, Draycott T, Winter C, Chew S, et al. How to be a very safe maternity unit: an ethnographic study. Soc Sci Med. 2019;223:64–72. https://doi.org/10.1016/j.socscimed.2019.01.035 .
doi: 10.1016/j.socscimed.2019.01.035
pubmed: 6391593
pmcid: 6391593
Draycott T. Practical obstetric multi-professional training – PROMPT. 2013. Available at: https://improve.bmj.com/sites/default/files/resources/prompt.pdf .
Richmond H, Copsey B, Hall AM, Davies D, Lamb SE. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ. 2017;17(1):227.
doi: 10.1186/s12909-017-1047-4
Mobasheri MH, King D, Johnston M, Gautama S, Purkayastha S, Darzi A. The ownership and clinical use of smartphones by doctors and nurses in the UK: a multicentre survey study. BMJ Innov. 2015;1(4):174–81. https://doi.org/10.1136/bmjinnov-2015-000062 .
doi: 10.1136/bmjinnov-2015-000062
Gosbee J. Human factors engineering and patient safety. Qual Saf Health Care. 2002;11(4):352–4. https://doi.org/10.1136/qhc.11.4.352 .
doi: 10.1136/qhc.11.4.352
pubmed: 1758019
pmcid: 1758019
Fricker RD. Sampling methods for web and e-mail surveys. In: Fielding N, Lee RM, Blank G, editors. The SAGE handbook of online research methods. London: SAGE Publications Ltd; 2008. https://doi.org/10.4135/9780857020055.n11 .
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. https://doi.org/10.1186/1745-6215-13-132 .
doi: 10.1186/1745-6215-13-132
pubmed: 3472231
pmcid: 3472231
Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409.
doi: 10.1186/s13063-016-1479-x
Maclennan S, Kirkham J, Lam TBL, Williamson PR. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. J Clin Epidemiol. 2018;93:1–8. https://doi.org/10.1016/j.jclinepi.2017.09.024 .
doi: 10.1016/j.jclinepi.2017.09.024
The Healthcare Improvement Studies (THIS) Institute. COVID-19: managing an obstetric emergency (video). 2020. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/covid-19-managing-an-obstetric-emergency/ .
The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (Infographic). 2020. Available at: https://www.thisinstitute.cam.ac.uk/wp-content/uploads/2020/11/TI_Covid_Obstertric_Emergency_A4_Graphic.pdf .
The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (information sheet). 2020. Available at: https://mk0thisinstitut0r1om.kinstacdn.com/wp-content/uploads/2020/11/TI_Covid_Obstertric_Emergency_A4_Colour.pdf .
Catchpole K. Improving performance through human-centred reconfiguration of existing designs. BMJ Qual Saf. 2013;22(1):5–7. https://doi.org/10.1136/bmjqs-2012-001677 .
doi: 10.1136/bmjqs-2012-001677
National Institute for Health and Care Excellence (NICE). Infection prevention and control. 2014. Available at: https://www.nice.org.uk/guidance/qs61 .
Brogaard L, Kierkegaard O, Hvidman L, Jensen K, Musaeus P, Uldbjerg N, et al. The importance of non-technical performance for teams managing postpartum haemorrhage: video review of 99 obstetric teams. BJOG. 2019;126:1015.
Watson SI, Dixon-Woods M, Taylor CA, Wroe EB, Dunbar EL, Chilton PJ, et al. Revising ethical guidance for the evaluation of programmes and interventions not initiated by researchers. J Med Ethics. 2020;46(1):26–30. https://doi.org/10.1136/medethics-2018-105263 .
doi: 10.1136/medethics-2018-105263
Tredick CA, Lewison RL, Deutschman DH, Hunt TA, Gordon KL, Von Hendy P. A rubric to evaluate citizen-science programs for long-term ecological monitoring. BioScience. 2017;67(9):834–44. https://doi.org/10.1093/biosci/bix090 .
doi: 10.1093/biosci/bix090
Kieslinger B, Schäfer T, Heigl F, Dörler D, Richter A, Bonn A. Evaluating citizen science-towards an open framework. In: Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018.
Gordon T, Pease A. RT Delphi: an efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Chang. 2006;73(4):321–33. https://doi.org/10.1016/j.techfore.2005.09.005 .
Gnatzy T, Warth J, von der Gracht H, Darkow I-L. Validating an innovative real-time Delphi approach - a methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Chang. 2011;78(9):1681–94. https://doi.org/10.1016/j.techfore.2011.04.006 .
doi: 10.1016/j.techfore.2011.04.006
Gordon TJ. The real-time Delphi method. Futures Res Methodol Version. 2009;3:19.
Rowe G, Wright G. The Delphi technique: past, present, and future prospects — introduction to the special issue. Technol Forecast Soc Chang. 2011;78(9):1487–90. https://doi.org/10.1016/j.techfore.2011.09.002 .
doi: 10.1016/j.techfore.2011.09.002
Ponti M, Hillman T, Kullenberg C, Kasperowski D. Getting it Right or Being Top Rank: Games in Citizen Science. Citizen Science. Theory and Pract. 2018;3(1):1–12. https://doi.org/10.5334/cstp.101 .
Veugelers R, Gaakeer MI, Patka P, Huijsman R. Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response. BMC Med Res Methodol. 2020;20(1):156.
doi: 10.1186/s12874-020-01029-4
Hall L, Eccles M. Case study of an inter-professional and inter-organisational programme to adapt, implement and evaluate clinical guidelines in secondary care. Br J Clin Gov. 2000;5(2):72–82. https://doi.org/10.1108/14664100010343863 .
doi: 10.1108/14664100010343863
Cook DJ, Pulido JN, Thompson JE, Dearani JA, Ritter MJ, Hanson AC, et al. Standardized practice design with electronic support mechanisms for surgical process improvement: reducing mechanical ventilation time. Ann Surg. 2014;260(6):1011–5. https://doi.org/10.1097/SLA.0000000000000726 .
doi: 10.1097/SLA.0000000000000726
Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implement Sci. 2018;13(1):72.
doi: 10.1186/s13012-018-0763-4
Patey R, Flin R, Fletcher G, Maran N, Glavin R. Developing a Taxonomy of Anesthetists’ Nontechnical Skills (ANTS). In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005. https://pubmed.ncbi.nlm.nih.gov/21250033/ .
Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13(Suppl 1):i85–90. https://doi.org/10.1136/qshc.2004.010033 .
doi: 10.1136/qshc.2004.010033
pubmed: 15465961
pmcid: 15465961
Portela MC, Pronovost JP, Woodcock T, Carter P, Mary D-W. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24(5):325–36. https://doi.org/10.1136/bmjqs-2014-003620 .
doi: 10.1136/bmjqs-2014-003620
pubmed: 4413733
pmcid: 4413733