Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning.
Journal
Nature biotechnology
ISSN: 1546-1696
Titre abrégé: Nat Biotechnol
Pays: United States
ID NLM: 9604648
Informations de publication
Date de publication:
04 2022
04 2022
Historique:
received:
01
03
2021
accepted:
14
09
2021
pubmed:
20
11
2021
medline:
15
4
2022
entrez:
19
11
2021
Statut:
ppublish
Résumé
A principal challenge in the analysis of tissue imaging data is cell segmentation-the task of identifying the precise boundary of every cell in an image. To address this problem we constructed TissueNet, a dataset for training segmentation models that contains more than 1 million manually labeled cells, an order of magnitude more than all previously published segmentation training datasets. We used TissueNet to train Mesmer, a deep-learning-enabled segmentation algorithm. We demonstrated that Mesmer is more accurate than previous methods, generalizes to the full diversity of tissue types and imaging platforms in TissueNet, and achieves human-level performance. Mesmer enabled the automated extraction of key cellular features, such as subcellular localization of protein signal, which was challenging with previous approaches. We then adapted Mesmer to harness cell lineage information in highly multiplexed datasets and used this enhanced version to quantify cell morphology changes during human gestation. All code, data and models are released as a community resource.
Identifiants
pubmed: 34795433
doi: 10.1038/s41587-021-01094-0
pii: 10.1038/s41587-021-01094-0
pmc: PMC9010346
mid: NIHMS1740588
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.
Research Support, N.I.H., Extramural
Langues
eng
Sous-ensembles de citation
IM
Pagination
555-565Subventions
Organisme : NCI NIH HHS
ID : P30 CA008748
Pays : United States
Organisme : NCI NIH HHS
ID : UH3 CA246633
Pays : United States
Organisme : NIA NIH HHS
ID : R01 AG056287
Pays : United States
Organisme : NIA NIH HHS
ID : R01 AG057915
Pays : United States
Organisme : NCI NIH HHS
ID : F31 CA246880
Pays : United States
Organisme : NIH HHS
ID : DP5 OD019822
Pays : United States
Informations de copyright
© 2021. The Author(s), under exclusive licence to Springer Nature America, Inc.
Références
Giesen, C. et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11, 417–422 (2014).
doi: 10.1038/nmeth.2869
pubmed: 24584193
Keren, L. et al. MIBI-TOF: a multiplexed imaging platform relates cellular phenotypes and tissue structure. Sci. Adv. 5, eaax5851 (2019).
doi: 10.1126/sciadv.aax5851
pubmed: 31633026
pmcid: 6785247
Huang, W., Hennrick, K. & Drew, S. A colorful future of quantitative pathology: validation of Vectra technology using chromogenic multiplexed immunohistochemistry and prostate tissue microarrays. Hum. Pathol. 44, 29–38 (2013).
doi: 10.1016/j.humpath.2012.05.009
pubmed: 22944297
Lin, J.-R. et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. eLife 7, e31657 (2018).
doi: 10.7554/eLife.31657
pubmed: 29993362
pmcid: 6075866
Gerdes, M. J. et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc. Natl Acad. Sci. 110, 11982–11987 (2013).
doi: 10.1073/pnas.1300136110
pubmed: 23818604
pmcid: 3718135
Goltsev, Y. et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell 174, 968–981.e15 (2018).
doi: 10.1016/j.cell.2018.07.010
pubmed: 30078711
pmcid: 6086938
Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015).
doi: 10.1126/science.aaa6090
pubmed: 25858977
pmcid: 4662681
Lee, J. H. et al. Highly multiplexed subcellular RNA sequencing in situ. Science 343, 1360–1363 (2014).
doi: 10.1126/science.1250212
pubmed: 24578530
pmcid: 4140943
Moffitt, J. R. et al. Molecular, spatial and functional single-cell profiling of the hypothalamic preoptic region. Science 362, eaau5324 (2018).
doi: 10.1126/science.aau5324
pubmed: 30385464
pmcid: 6482113
Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science 361, eaat5691 (2018).
doi: 10.1126/science.aat5691
pubmed: 29930089
pmcid: 6339868
Lubeck, E., Coskun, A. F., Zhiyentayev, T., Ahmad, M. & Cai, L. Single-cell in situ RNA profiling by sequential hybridization. Nat Methods 11, 360–361 (2014).
doi: 10.1038/nmeth.2892
pubmed: 24681720
pmcid: 4085791
Eng, C.-H. L. et al. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+. Nature 568, 235–239 (2019).
doi: 10.1038/s41586-019-1049-y
pubmed: 30911168
pmcid: 6544023
Rozenblatt-Rosen, O. et al. The human tumor atlas network: charting tumor transitions across space and time at single-cell resolution. Cell 181, 236–249 (2020).
doi: 10.1016/j.cell.2020.03.053
pubmed: 32302568
pmcid: 7376497
Snyder, M. P. et al. The human body at cellular resolution: the NIH Human Biomolecular Atlas Program. Nature 574, 187–192 (2019).
doi: 10.1038/s41586-019-1629-x
Regev, A. et al. The human cell atlas white paper. Preprint at https://arxiv.org/abs/1810.05192v1 (2018).
Keren, L. et al. A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging. Cell 174, 1373–1387.e19 (2018).
doi: 10.1016/j.cell.2018.08.039
pubmed: 30193111
pmcid: 6132072
Milo, R. & Phillips, R. Cell Biology by the Numbers 1st edn (Garland Science, 2015).
Mescher, A. Junqueira’s Basic Histology: Text and Atlas 13th edn (McGraw Hill, 2013).
McQuin, C. et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).
doi: 10.1371/journal.pbio.2005970
pubmed: 29969450
pmcid: 6029841
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
doi: 10.1038/nmeth.2019
pubmed: 22743772
Schneider, C. A., Rasband, W. S. & Eliceiri, K. W. NIH Image to ImageJ: 25 years of image analysis. Nat. Methods 9, 671–675 (2012).
doi: 10.1038/nmeth.2089
pubmed: 22930834
pmcid: 5554542
Berg, S. et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).
doi: 10.1038/s41592-019-0582-9
pubmed: 31570887
de Chaumont, F. Icy: an open bioimage informatics platform for extended reproducible research. Nat. Methods 9, 690–696 (2012).
doi: 10.1038/nmeth.2075
pubmed: 22743774
Belevich, I., Joensuu, M., Kumar, D., Vihinen, H. & Jokitalo, E. Microscopy image browser: a platform for segmentation and analysis of multidimensional datasets. PLoS Biol. 14, e1002340 (2016).
doi: 10.1371/journal.pbio.1002340
pubmed: 26727152
pmcid: 4699692
Ronneberger, O., Fischer, P. & Brox, T. in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 (eds Navab, N. et al.) 234–241 (Lecture Notes in Computer Science 9351, Springer, 2015).
Valen, D. A. V. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 12, e1005177 (2016).
doi: 10.1371/journal.pcbi.1005177
pubmed: 27814364
pmcid: 5096676
Caicedo, J. C. et al. Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl. Nat. Methods 16, 1247–1253 (2019).
doi: 10.1038/s41592-019-0612-7
pubmed: 31636459
pmcid: 6919559
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).
doi: 10.1038/s41592-020-01018-x
pubmed: 33318659
Hollandi, R. et al. nucleAIzer: a parameter-free deep learning framework for nucleus segmentation using image style transfer. Cell Syst. 10, 453–458.e6 (2020).
doi: 10.1016/j.cels.2020.04.003
pubmed: 34222682
pmcid: 8247631
Koyuncu, C. F., Gunesli, G. N., Cetin-Atalay, R. & Gunduz-Demir, C. DeepDistance: a multi-task deep regression model for cell detection in inverted microscopy images. Med. Image Anal. 63, 101720 (2020).
doi: 10.1016/j.media.2020.101720
pubmed: 32438298
Yang, L. et al. NuSeT: A deep learning tool for reliably separating and analyzing crowded cells. PLoS Comput. Biol. 16, e1008193 (2020).
doi: 10.1371/journal.pcbi.1008193
pubmed: 32925919
pmcid: 7515182
Yu, W. et al. CCDB:6843, mus musculus, Neuroblastoma. CIL. Dataset. https://doi.org/10.7295/W9CCDB6843
Koyuncu, C. F., Cetin‐Atalay, R. & Gunduz‐Demir, C. Object‐oriented segmentation of cell nuclei in fluorescence microscopy images. Cytometry A 93, 1019–1028 (2018).
doi: 10.1002/cyto.a.23594
pubmed: 30211975
Ljosa, V., Sokolnicki, K. L. & Carpenter, A. E. Annotated high-throughput microscopy image sets for validation. Nat. Methods 9, 637–637 (2012).
doi: 10.1038/nmeth.2083
pubmed: 22743765
pmcid: 3627348
Kumar, N. et al. A multi-organ nucleus segmentation challenge. IEEE Trans. Med. Imaging 39, 1380–1391 (2020).
doi: 10.1109/TMI.2019.2947628
pubmed: 31647422
Verma, R. et al. MoNuSAC2020: A Multi-organ Nuclei Segmentation and Classification Challenge. IEEE Trans. Med. Imaging 10.1109/TMI.2021.3085712 (2021).
Moen, E. et al. Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. Preprint at bioRxiv https://doi.org/10.1101/803205 (2019).
Gamper, J. et al. PanNuke dataset extension, insights and baselines. Preprint at https://arxiv.org/abs/2003.10778v7 (2020).
Bannon, D. et al. DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes. Nat. Methods 18, 43–45 (2021).
doi: 10.1038/s41592-020-01023-0
pubmed: 33398191
pmcid: 8759612
Haberl, M. G. et al. CDeep3M—plug-and-play cloud-based deep learning for image segmentation. Nat. Methods 15, 677–680 (2018).
doi: 10.1038/s41592-018-0106-z
pubmed: 30171236
pmcid: 6548193
Ouyang, W., Mueller, F., Hjelmare, M., Lundberg, E. & Zimmer, C. ImJoy: an open-source computational platform for the deep learning era. Nat. Methods 16, 1199–1200 (2019).
doi: 10.1038/s41592-019-0627-0
pubmed: 31780825
von Chamier, L. et al. Democratising deep learning for microscopy with ZeroCostDL4Mic. Nat. Commun. 12, 2276 (2021).
doi: 10.1038/s41467-021-22518-0
pubmed: 33859193
pmcid: 8050272
Hughes, A. J. et al. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat. Methods 15, 587–590 (2018).
doi: 10.1038/s41592-018-0069-0
pubmed: 30065368
pmcid: 8863499
Ouyang, W., Le, T., Xu, H. & Lundberg, E. Interactive biomedical segmentation tool powered by deep learning and ImJoy. F1000Research 10, 142 (2021).
doi: 10.12688/f1000research.50798.1
Wolny, A. et al. Accurate and versatile 3D segmentation of plant tissues at cellular resolution. eLife 9, e57613 (2020).
doi: 10.7554/eLife.57613
pubmed: 32723478
pmcid: 7447435
DeepCell Label: https://github.com/vanvalenlab/deepcell-label
Lin, T.-Y. et al. Feature pyramid networks for object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2117–2125 (IEEE, 2017).
Tan, M., Pang, R. & Le, Q. V. EfficientDet: scalable and efficient object detection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 10778–10787 (IEEE, 2020).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, 2016).
Zuiderveld, K. in Graphics Gems (ed Heckbert, P. S.) Ch. VIII.5 (Academic Press, 1994).
Chevalier, G. Make smooth predictions by blending image patches, such as for image segmentation. https://github.com/Vooban/Smoothly-Blend-Image-Patches
Meyer, F. & Beucher, S. Morphological segmentation. J. Vis. Commun. Image R 1, 21–46 (1990).
doi: 10.1016/1047-3203(90)90014-M
Weigert, M., Schmidt, U., Haase, R., Sugawara, K. & Myers, G. Star-convex polyhedra for 3D object detection and segmentation in microscopy. In IEEE Winter Conference on Applications of Computer Vision (WACV) 3655–3662 (IEEE, 2020).
Fu, C.-Y., Shvets, M. & Berg, A. C. RetinaMask: learning to predict masks improves state-of-the-art single-shot detection for free. Preprint at https://arxiv.org/abs/1901.03353v1 (2019).
Schürch, C. M. et al. Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell 182, 1341–1359.e19 (2020).
doi: 10.1016/j.cell.2020.07.005
pubmed: 32763154
pmcid: 7479520
Ali, H. R. et al. Imaging mass cytometry and multiplatform genomics define the phenogenomic landscape of breast cancer. Nat. Cancer 1, 163–175 (2020).
doi: 10.1038/s43018-020-0026-6
pubmed: 35122013
Gaglia, G. et al. HSF1 phase transition mediates stress adaptation and cell fate decisions. Nat. Cell Biol. 22, 151–158 (2020).
doi: 10.1038/s41556-019-0458-3
pubmed: 32015439
pmcid: 7135912
Nelson, D. E. et al. Oscillations in NF-κB signaling control the dynamics of gene expression. Science 306, 704–708 (2004).
doi: 10.1126/science.1099962
pubmed: 15499023
Kumar, K. P., McBride, K. M., Weaver, B. K., Dingwall, C. & Reich, N. C. Regulated nuclear-cytoplasmic localization of interferon regulatory factor 3, a subunit of double-stranded RNA-activated factor 1. Mol. Cell Biol. 20, 4159–4168 (2000).
doi: 10.1128/MCB.20.11.4159-4168.2000
pubmed: 10805757
pmcid: 85785
Wolff, A. C. et al. Recommendations for human epidermal growth factor receptor 2 testing in Breast Cancer: American Society of Clinical Oncology/College of American pathologists clinical practice guideline update. J. Clin. Oncol. 31, 3997–4013 (2013).
doi: 10.1200/JCO.2013.50.9984
pubmed: 24101045
Risom, T. et al. Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma. Preprint at bioRxiv https://doi.org/10.1101/2021.01.05.425362 (2021)
Ark Analysis. https://github.com/angelolab/ark-analysis
Koss, L. G. Diagnostic Cytology and Its Histopathologic Bases. (J.B. Lippincott Company, 1979).
Erlebacher, A. Immunology of the maternal-fetal interface. Annu. Rev. Immunol. 31, 387–411 (2013).
doi: 10.1146/annurev-immunol-032712-100003
pubmed: 23298207
Greenbaum, S. et al. Spatio-temporal coordination at the maternal-fetal interface promotes trophoblast invasion and vascular remodeling in the first half of human pregnancy. Preprint at bioRxiv https://doi.org/10.1101/2021.09.08.459490 (2021).
Garrido-Gomez, T. et al. Defective decidualization during and after severe preeclampsia reveals a possible maternal contribution to the etiology. Proc. Natl Acad. Sci. USA 114, E8468–E8477 (2017).
doi: 10.1073/pnas.1706546114
pubmed: 28923940
pmcid: 5635883
Deep Cell Core Library. Deep learning for single-cell analysis. https://github.com/vanvalenlab/deepcell-tf
Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).
doi: 10.1038/s41598-017-17204-5
pubmed: 29203879
pmcid: 5715110
Graham, S. et al. Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019).
doi: 10.1016/j.media.2019.101563
pubmed: 31561183
Tsai, H.-F., Gajda, J., Sloan, T. F. W., Rares, A. & Shen, A. Q. Usiigaci: instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning. SoftwareX 9, 230–237 (2019).
doi: 10.1016/j.softx.2019.02.007
Kiemen, A. et al. In situ characterization of the 3D microanatomy of the pancreas and pancreatic cancer at single cell resolution. bioRxiv 2020.12.08.416909 (2020) https://doi.org/10.1101/2020.12.08.416909
Cao, J. et al. Establishment of a morphological atlas of the Caenorhabditis elegans embryo using deep-learning-based 4D segmentation. Nat. Commun. 11, 6254 (2020).
doi: 10.1038/s41467-020-19863-x
pubmed: 33288755
pmcid: 7721714
Schulz, D. et al. Simultaneous multiplexed imaging of mRNA and proteins with subcellular resolution in breast cancer tissue samples by mass cytometry. Cell Syst. 6, 531 (2018).
doi: 10.1016/j.cels.2018.04.004
pubmed: 29698648
pmcid: 5929910
McKinley, E. T. et al. Optimized multiplex immunofluorescence single-cell analysis reveals tuft cell heterogeneity. JCI Insight 2, e93487 (2017).
doi: 10.1172/jci.insight.93487
pmcid: 5453701
Patel, S. S. et al. The microenvironmental niche in classic Hodgkin lymphoma is enriched for CTLA-4- positive T-cells that are PD-1-negative. Blood 134, 2059–2069 (2019).
doi: 10.1182/blood-2019-122535
pubmed: 31697809
pmcid: 7218752
Jackson, H. W. et al. The single-cell pathology landscape of breast cancer. Nature 578, 615–620 (2020).
doi: 10.1038/s41586-019-1876-x
pubmed: 31959985
Rashid, R. et al. Highly multiplexed immunofluorescence images and single-cell data of immune markers in tonsil and lung cancer. Sci. Data 6, 323 (2019).
doi: 10.1038/s41597-019-0332-y
pubmed: 31848351
pmcid: 6917801
McCaffrey, E. F. et al. Multiplexed imaging of human tuberculosis granulomas uncovers immunoregulatory features conserved across tissue and blood. Preprint at bioRxiv https://doi.org/10.1101/2020.06.08.140426 (2020).
Walt, Svander et al. scikit-image: image processing in Python. PeerJ 2, e453 (2014).
doi: 10.7717/peerj.453
pubmed: 25024921
pmcid: 4081273
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980v9 (2014).
Kluyver, T. et al. in Positioning and Power in Academic Publishing: Players, Agents and Agendas (eds Schmidt, B. & Loizides, F.) (IOS Press, 2016).
Chollet, F. et al. Keras. https://keras.io (2015).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
doi: 10.1109/MCSE.2007.55
Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020).
doi: 10.1038/s41586-020-2649-2
pubmed: 32939066
pmcid: 7759461
Reback, J. et al. pandas-dev/pandas: Pandas 1.1.3. https://doi.org/10.5281/zenodo.3509134 (2020).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. Preprint at https://arxiv.org/abs/1201.0490v4 (2012).
Waskom, M. et al. mwaskom/seaborn. https://doi.org/10.5281/zenodo.592845 (2020).
Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467v2 (2016).
Hoyer, S. & Hamman, J. xarray: N-D labeled arrays and datasets in Python. J. Open Res. Softw. 5, 10 (2017).
doi: 10.5334/jors.148