Comparison of humans versus mobile phone-powered artificial intelligence for the diagnosis and management of pigmented skin cancer in secondary care: a multicentre, prospective, diagnostic, clinical trial.

BACKGROUND

Diagnosis of skin cancer requires medical expertise, which is scarce. Mobile phone-powered artificial intelligence (AI) could aid diagnosis, but it is unclear how this technology performs in a clinical scenario. Our primary aim was to test in the clinic whether there was equivalence between AI algorithms and clinicians for the diagnosis and management of pigmented skin lesions.

METHODS

In this multicentre, prospective, diagnostic, clinical trial, we included specialist and novice clinicians and patients from two tertiary referral centres in Australia and Austria. Specialists had a specialist medical qualification related to diagnosing and managing pigmented skin lesions, whereas novices were dermatology junior doctors or registrars in trainee positions who had experience in examining and managing these lesions. Eligible patients were aged 18-99 years and had a modified Fitzpatrick I-III skin type; those in the diagnostic trial were undergoing routine excision or biopsy of one or more suspicious pigmented skin lesions bigger than 3 mm in the longest diameter, and those in the management trial had baseline total-body photographs taken within 1-4 years. We used two mobile phone-powered AI instruments incorporating a simple optical attachment: a new 7-class AI algorithm and the International Skin Imaging Collaboration (ISIC) AI algorithm, which was previously tested in a large online reader study. The reference standard for excised lesions in the diagnostic trial was histopathological examination; in the management trial, the reference standard was a descending hierarchy based on histopathological examination, comparison of baseline total-body photographs, digital monitoring, and telediagnosis. The main outcome of this study was to compare the accuracy of expert and novice diagnostic and management decisions with the two AI instruments. Possible decisions in the management trial were dismissal, biopsy, or 3-month monitoring. Decisions to monitor were considered equivalent to dismissal (scenario A) or biopsy of malignant lesions (scenario B). The trial was registered at the Australian New Zealand Clinical Trials Registry ACTRN12620000695909 (Universal trial number U1111-1251-8995).

FINDINGS

The diagnostic study included 172 suspicious pigmented lesions (84 malignant) from 124 patients and the management study included 5696 pigmented lesions (18 malignant) from the whole body of 66 high-risk patients. The diagnoses of the 7-class AI algorithm were equivalent to the specialists' diagnoses (absolute accuracy difference 1·2% [95% CI -6·9 to 9·2]) and significantly superior to the novices' ones (21·5% [13·1 to 30·0]). The diagnoses of the ISIC AI algorithm were significantly inferior to the specialists' diagnoses (-11·6% [-20·3 to -3·0]) but significantly superior to the novices' ones (8·7% [-0·5 to 18·0]). The best 7-class management AI was significantly inferior to specialists' management (absolute accuracy difference in correct management decision -0·5% [95% CI -0·7 to -0·2] in scenario A and -0·4% [-0·8 to -0·05] in scenario B). Compared with the novices' management, the 7-class management AI was significantly inferior (-0·4% [-0·6 to -0·2]) in scenario A but significantly superior (0·4% [0·0 to 0·9]) in scenario B.

INTERPRETATION

The mobile phone-powered AI technology is simple, practical, and accurate for the diagnosis of suspicious pigmented skin cancer in patients presenting to a specialist setting, although its usage for management decisions requires more careful execution. An AI algorithm that was superior in experimental studies was significantly inferior to specialists in a real-world scenario, suggesting that caution is needed when extrapolating results of experimental studies to clinical practice.

FUNDING

MetaOptima Technology.

Informations de copyright

Copyright © 2023 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license. Published by Elsevier Ltd.. All rights reserved.

Déclaration de conflit d'intérêts

Declaration of interests SWM is on the scientific advisory board of SciBase AB, and previously for Kāhu (MoleMap), both of which produce competing diagnostic devices for skin cancer. MM received payment from MetaOptima Technology for doing the trial in Sydney. She has also received payments from MetaOptima Technology (unrelated to and before the trial) as a consultant. SNL's institute has received payment for consultancy services from MetaOptima Technology, unrelated to the trial. WY, JL, and MR are employees of MetaOptima Technology. MR is a board member and equity owner of MetaOptima Technology. PT has received fees for professional services from Silverchair; an unrestricted 1-year postdoctoral grant from MetaOptima Technology (2017); speaker fees from Novartis, Lilly, and FotoFinder; and unrestricted grants from Lilly. He is also on the executive board of the International Dermoscopy Society, is President of the Austrian Dermatopathology Society, and is a coauthor of textbooks on dermatoscopy (published by Facultas) and dermatopathology (published by Springer). PG has received fees for professional services from MetaOptima Technology, unrelated to the trial. She has also received fees from La Roche-Posay and Pierre Fabre, support for attending meetings from La Roche-Posay, and is on the faculty of Melanoma Institute Australia. RAS has received fees for professional services from MetaOptima Technology, F Hoffmann-La Roche, Evaxion Biotech, Provectus Biopharmaceuticals Australia, Qbiotics, Novartis, Merck Sharp & Dohme, NeraCare, Amgen, Bristol Myers Squibb, Myriad Genetics, and GlaxoSmithKline. HHC serves as a medical advisor to Metasense. DC has served as a speaker for AbbVie, Gilead Sciences, ViiV Healthcare, and MSD; has served as an advisory board member for ViiV Healthcare; and has received travel support from AbbVie, MSD, ViiV Healthcare, and Gilead Sciences. MG has received a conference travel grant from Sun Pharmaceutical Industries. GH has been supported by an Avant Foundation Early Career Researcher grant and by National Health and Medical Research Council Centres of Research Excellence funding (1135285). PK has received fees and travel support from Sanofi. JMR has received speakers honoraria from Bristol Myers Squibb, Roche, Amgen, and Novartis, and travel support from Roche, Bristol Myers Squibb, and Sanofi. MS has received consulting fees and travel support from Gilead Sciences, ViiV Healthcare/GSK, and MSD; is on the advisory boards of Gilead Sciences, ViiV Healthcare/GSK, and MSD; and is a board member of the Austrian Sexually Transmitted Disease Society. JT has received speaker fees from Lilly and Novartis, travel support from Almirall and AbbVie, and an unrestricted grant from Lilly. PW has received travel support from Pfizer and Merz Pharma, and fees for a case report from Novartis. WW has received a grant from WWTF; consulting fees from Böhringer Ingelheim, Sanofi Genzyme, Eli Lilly, AbbVie, LEO Pharma, Janssen, and Pfizer. He is a committee member of the Austrian Society of Dermatology and Venerology and is on the advisory boards of Sanofi Genzyme, Böhringer Ingelheim, and Eli Lilly. CZ was a coinvestigator on Sydney Cancer Partners Pilot Study Scheme Grant Procel Study-Perioperative Propranolol and Celexocib in Stage III melanoma. HK has received speaker fees from La Roche-Posay, Novartis, Eli Lilly, Pelpharma, Heine, and FotoFinder; has received equipment to his institution from Heine, FotoFinder, Canfield, and DermaMedical; and has received license fees to his institution from MetaOptima Technology, Heine, and Casi. He is member of the executive board of the International Dermoscopy Society. All other authors declare no competing interests.