Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology.
Journal
Modern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc
ISSN: 1530-0285
Titre abrégé: Mod Pathol
Pays: United States
ID NLM: 8806605
Informations de publication
Date de publication:
12 2022
12 2022
Historique:
received:
06
05
2022
accepted:
25
07
2022
revised:
24
07
2022
pubmed:
11
9
2022
medline:
2
12
2022
entrez:
10
9
2022
Statut:
ppublish
Résumé
Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.
Identifiants
pubmed: 36088478
doi: 10.1038/s41379-022-01147-y
pii: S0893-3952(22)05487-4
pmc: PMC9708586
doi:
Types de publication
Journal Article
Review
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
1759-1769Commentaires et corrections
Type : ErratumIn
Informations de copyright
© 2022. The Author(s).
Références
J Clin Epidemiol. 2005 Aug;58(8):859-62
pubmed: 16018921
Med Image Anal. 2022 Apr;77:102337
pubmed: 35016078
Stat Med. 2007 Dec 30;26(30):5499-511
pubmed: 17907249
Front Med (Lausanne). 2019 Sep 30;6:193
pubmed: 31632974
Mod Pathol. 2021 Dec;34(12):2098-2108
pubmed: 34168282
Nat Med. 2018 Oct;24(10):1559-1567
pubmed: 30224757
Circ Cardiovasc Qual Outcomes. 2020 Oct;13(10):e006556
pubmed: 33079589
Genome Biol. 2005;6(5):R47
pubmed: 15892875
ESMO Open. 2022 Apr;7(2):100400
pubmed: 35247870
Turk Patoloji Derg. 2016;32(2):112-8
pubmed: 27136110
J Pathol Inform. 2018 Nov 02;9:37
pubmed: 30533276
J R Coll Physicians Edinb. 2020 Dec;50(4):398-402
pubmed: 33469615
Med Image Anal. 2022 Jan;75:102274
pubmed: 34731777
J Oral Maxillofac Pathol. 2014 Sep;18(Suppl 1):S111-6
pubmed: 25364159
Diagn Pathol. 2017 Nov 13;12(1):80
pubmed: 29132399
Histopathology. 2014 Aug;65(2):240-51
pubmed: 24511905
Med Image Anal. 2019 Dec;58:101544
pubmed: 31466046
Nat Med. 2019 Jul;25(7):1054-1056
pubmed: 31160815
Neuroimaging Clin N Am. 2020 Nov;30(4):433-445
pubmed: 33038994
J Pathol Inform. 2021 Nov 03;12:43
pubmed: 34881098
IEEE Trans Med Imaging. 2019 Oct;38(10):2293-2302
pubmed: 30762541
BMJ Open. 2021 Jul 9;11(7):e048008
pubmed: 34244270
Stat Med. 2021 Aug 30;40(19):4230-4251
pubmed: 34031906
Proc ACM Conf Health Inference Learn (2020). 2020 Apr;2020:151-159
pubmed: 33196064
J Pathol Inform. 2021 Dec 24;12:54
pubmed: 35070483
Mod Pathol. 2021 Aug;34(8):1588-1595
pubmed: 33782551
Comput Biol Med. 2022 Mar;142:105219
pubmed: 35026572
Eval Health Prof. 2003 Sep;26(3):258-87
pubmed: 12971200
Hepatology. 2020 Dec;72(6):2000-2013
pubmed: 32108950
BMC Med Inform Decis Mak. 2020 Sep 11;20(1):219
pubmed: 32917183
Sci Rep. 2020 Feb 21;10(1):3217
pubmed: 32081956
CA Cancer J Clin. 2021 May;71(3):209-249
pubmed: 33538338
J Pathol Inform. 2021 Aug 04;12:30
pubmed: 34497734
Nat Med. 2019 Aug;25(8):1301-1309
pubmed: 31308507
BMJ. 2021 Sep 1;374:n1872
pubmed: 34470740
Pathologe. 2021 Dec;42(Suppl 2):135-141
pubmed: 34919184
J Clin Epidemiol. 1991;44(8):763-70
pubmed: 1941027
Lancet. 2020 Feb 1;395(10221):350-360
pubmed: 32007170
Surg Pathol Clin. 2020 Jun;13(2):349-358
pubmed: 32389272
BMC Med. 2019 Oct 29;17(1):195
pubmed: 31665002
IEEE Trans Med Imaging. 2007 Jul;26(7):1010-6
pubmed: 17649914
Front Med (Lausanne). 2019 Oct 01;6:185
pubmed: 31632973
Br J Cancer. 2021 Feb;124(4):686-696
pubmed: 33204028
J Pathol. 2019 Nov;249(3):286-294
pubmed: 31355445
Lancet. 2019 Jul 6;394(10192):9-11
pubmed: 30935732
JAMA. 2020 Sep 22;324(12):1212-1213
pubmed: 32960230
Oral Oncol. 2007 Mar;43(3):224-31
pubmed: 16931119
J Med Internet Res. 2021 Feb 2;23(2):e23436
pubmed: 33528370
J Pathol. 2020 Oct;252(2):178-188
pubmed: 32686118
J Pathol Inform. 2017 Apr 10;8:19
pubmed: 28480122
Ann Intern Med. 2015 Jan 6;162(1):W1-73
pubmed: 25560730
Sci Rep. 2017 Apr 18;7:46450
pubmed: 28418027
PLoS One. 2018 Oct 25;13(10):e0205387
pubmed: 30359393
J Pathol. 2021 Mar;253(3):268-278
pubmed: 33197281
J Pathol Inform. 2021 Nov 15;12:45
pubmed: 34881099
Nat Rev Clin Oncol. 2018 Feb;15(2):81-94
pubmed: 29115304
Lancet Digit Health. 2022 Jan;4(1):e18-e26
pubmed: 34794930
NPJ Breast Cancer. 2018 Sep 3;4:30
pubmed: 30182055
Acta Oncol. 2000;39(1):41-5
pubmed: 10752652
Breast. 2021 Apr;56:78-87
pubmed: 33640523
Radiology. 1982 Apr;143(1):29-36
pubmed: 7063747
BMJ. 2020 Sep 9;370:m3164
pubmed: 32909959
IEEE J Biomed Health Inform. 2021 Feb;25(2):325-336
pubmed: 33085623
Korean J Radiol. 2021 Mar;22(3):442-453
pubmed: 33629545
Nat Commun. 2021 Jul 20;12(1):4423
pubmed: 34285218
BMJ. 2009 Oct 06;339:b3985
pubmed: 19808754
JAMA Intern Med. 2018 Nov 1;178(11):1544-1547
pubmed: 30128552
J Pathol Inform. 2016 Jul 26;7:29
pubmed: 27563488
J Oral Maxillofac Pathol. 2018 May-Aug;22(2):279
pubmed: 30158787
Stat Methods Med Res. 2021 Oct;30(10):2187-2206
pubmed: 33881369
NPJ Digit Med. 2022 Jan 10;5(1):2
pubmed: 35013569
PLoS One. 2018 May 10;13(5):e0196547
pubmed: 29746503
Comput Med Imaging Graph. 2018 Dec;70:43-52
pubmed: 30286333
Nat Med. 2020 Sep;26(9):1320-1324
pubmed: 32908275
Cytometry A. 2016 Dec;89(12):1057-1072
pubmed: 27922735
Lupus. 2017 Oct;26(11):1205-1211
pubmed: 28478696
Eur J Cancer. 2017 Oct;84:219-227
pubmed: 28829990
Int J Comput Vis. 2008;79(3):225-230
pubmed: 18776952
PLoS One. 2015 Mar 04;10(3):e0118432
pubmed: 25738806
IEEE Trans Cybern. 2020 Sep;50(9):3950-3962
pubmed: 31484154
JCO Clin Cancer Inform. 2019 Apr;3:1-7
pubmed: 30990737
Med Image Anal. 2022 Apr;77:102368
pubmed: 35063892
Sci Rep. 2020 Jan 30;10(1):1504
pubmed: 32001752
Ann Surg. 2009 Nov;250(5):691-7
pubmed: 19806055
NPJ Digit Med. 2019 Jun 7;2:48
pubmed: 31304394
Am J Pathol. 2021 Oct;191(10):1702-1708
pubmed: 33636179
J Thorac Dis. 2017 Oct;9(10):4125-4130
pubmed: 29268424
Nat Med. 2021 Apr;27(4):582-584
pubmed: 33820998
Nat Med. 2022 Jan;28(1):154-163
pubmed: 35027755
J Mol Med (Berl). 2020 Feb;98(2):161-177
pubmed: 31970428
JAMIA Open. 2021 Sep 24;4(3):ooab077
pubmed: 34568771
Arch Pathol Lab Med. 2007 Jul;131(7):1084-8
pubmed: 17616995
J Pathol Inform. 2021 Mar 22;12:13
pubmed: 34012717
Comput Methods Programs Biomed. 2021 Sep;208:106288
pubmed: 34352688
J Pathol. 2021 Jun;254(2):147-158
pubmed: 33904171
N Engl J Med. 2021 Jul 15;385(3):283-286
pubmed: 34260843
J Pathol Inform. 2021 Jan 23;12:5
pubmed: 34012709
BMJ Open. 2021 Jun 28;11(6):e047709
pubmed: 34183345
Am J Surg Pathol. 2003 Jun;27(6):805-10
pubmed: 12766585
Diagn Pathol. 2014;9 Suppl 1:S3
pubmed: 25565494