An intentional approach to managing bias in general purpose embedding models.

Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.

Informations de copyright

Copyright © 2024 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license. Published by Elsevier Ltd.. All rights reserved.

Déclaration de conflit d'intérêts

Declaration of interests W-HW, AS, APK, AD’A, JP, RP, SP, VN, SA, AK, HC-L, YM, GSC, DRW, SS, SP, KE, and YL are employees of Google and hold Alphabet stock. W-HW, AS, APK, AD’A, RP, SP, VN, SA, AK, HC-L, YM, GSC, DRW, SS, SP, KE, and YL have patents filed or in progress under Google, broadly related to machine learning and embedding models. CL performs work at Google as a medical consultant via Vituity and receives consulting fees for clinical perspective and guidance. LAGC receives support for educational events and meetings from the National Institutes of Health, Stanford University, University of California San Francisco, University of Toronto, College of Intensive Care Medicine of Australia and New Zealand, University of Bergen, Amsterdam University Medical Centers, Académie Nationale de Médecine (France), and the Doris Duke Foundation (for the Reconsidering Race in Clinical Algorithms workshop at the National Academy of Medicine, Washington, DC). LAGC is an Editor in Chief of PLOS Digital Health, and on the advisory board of the Lancet Digital Health.