Democratising deep learning for microscopy with ZeroCostDL4Mic.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
15 04 2021
15 04 2021
Historique:
received:
12
08
2020
accepted:
10
03
2021
entrez:
16
4
2021
pubmed:
17
4
2021
medline:
4
5
2021
Statut:
epublish
Résumé
Deep Learning (DL) methods are powerful analytical tools for microscopy and can outperform conventional image processing pipelines. Despite the enthusiasm and innovations fuelled by DL technology, the need to access powerful and compatible resources to train DL networks leads to an accessibility barrier that novice users often find difficult to overcome. Here, we present ZeroCostDL4Mic, an entry-level platform simplifying DL access by leveraging the free, cloud-based computational resources of Google Colab. ZeroCostDL4Mic allows researchers with no coding expertise to train and apply key DL networks to perform tasks including segmentation (using U-Net and StarDist), object detection (using YOLOv2), denoising (using CARE and Noise2Void), super-resolution microscopy (using Deep-STORM), and image-to-image translation (using Label-free prediction - fnet, pix2pix and CycleGAN). Importantly, we provide suitable quantitative tools for each network to evaluate model performance, allowing model optimisation. We demonstrate the application of the platform to study multiple biological processes.
Identifiants
pubmed: 33859193
doi: 10.1038/s41467-021-22518-0
pii: 10.1038/s41467-021-22518-0
pmc: PMC8050272
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
2276Subventions
Organisme : Medical Research Council
ID : MR/K015826/1
Pays : United Kingdom
Organisme : Medical Research Council
ID : MR/T027924/1
Pays : United Kingdom
Organisme : Wellcome Trust
Pays : United Kingdom
Organisme : Medical Research Council
ID : MC_UU_00012/1
Pays : United Kingdom
Organisme : Cancer Research UK
ID : FC001999
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 203276/Z/16/Z
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 206670/Z/17/Z
Pays : United Kingdom
Organisme : Medical Research Council
ID : FC001999
Pays : United Kingdom
Références
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. in Advances in Neural Information Processing Systems 25 (eds. Pereira, F. et al.) 1097–1105 (Curran Associates, Inc., 2012).
Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. pp. 234–241 (Springer, Cham, 2015).
Redmon, J. & Farhadi, A. YOLO9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7263–7271 (2017).
Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).
pubmed: 28778026
doi: 10.1016/j.media.2017.07.005
Schmidt, U., Weigert, M., Broaddus, C. & Myers, G. Cell detection with Star-Convex polygons. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds. Frangi, A. F. et al.) Vol. 11071, 265–273 (Springer International Publishing, 2018).
Weigert, M., Schmidt, U., Haase, R., Sugawara, K. & Myers, G. Star-convex polyhedra for 3d object detection and segmentation in microscopy. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3666–3673 (2020).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).
pubmed: 30478326
doi: 10.1038/s41592-018-0216-7
Krull, A., Buchholz, T. O. & Jug, F. Noise2void-learning denoising from single noisy images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2129–2137 (2019).
Araújo, T. et al. Classification of breast cancer histology images using convolutional neural networks. PLoS ONE 12, e0177544 (2017).
pubmed: 28570557
pmcid: 5453426
doi: 10.1371/journal.pone.0177544
Haberl, M. G. et al. CDeep3M—Plug-and-Play cloud-based deep learning for image segmentation. Nat. Methods 15, 677–680 (2018).
pubmed: 30171236
pmcid: 6548193
doi: 10.1038/s41592-018-0106-z
Buchholz, T. O., Prakash, M., Schmidt, D., Krull, A. & Jug, F. DenoiSeg: joint denoising and segmentation. European Conference on Computer Vision. pp. 324-337 (Springer, Cham, 2020).
Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 12, e1005177 (2016).
pubmed: 27814364
pmcid: 5096676
doi: 10.1371/journal.pcbi.1005177
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
pubmed: 26017442
doi: 10.1038/nature14539
Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).
pubmed: 30559429
doi: 10.1038/s41592-018-0261-2
Bannon, D. et al. DeepCell Kiosk: scaling deep learning-enabled cellular image analysis with Kubernetes. Nat. Methods 18, 43–5 (2021).
pubmed: 33398191
doi: 10.1038/s41592-020-01023-0
pmcid: 8759612
Belevich, I. & Jokitalo, E. DeepMIB: user-friendly and open-source software for training of deep learning network for biological image segmentation. PLoS Comput Biol. 17, e1008374 (2021).
pubmed: 33651804
pmcid: 7954287
doi: 10.1371/journal.pcbi.1008374
Hollandi, R., Szkalisity, A. & Toth, T. nucleAIzer: a parameter-free deep learning framework for nucleus segmentation using image style transfer. Cell Syst. 10, 453–458 (2020).
doi: 10.1016/j.cels.2020.04.003
pubmed: 34222682
pmcid: 8247631
Ouyang, W., Mueller, F., Hjelmare, M., Lundberg, E. & Zimmer, C. ImJoy: an open-source computational platform for the deep learning era. Nat. Methods 16, 1199–1200 (2019).
pubmed: 31780825
doi: 10.1038/s41592-019-0627-0
Berg, S. et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).
pubmed: 31570887
doi: 10.1038/s41592-019-0582-9
McQuin, C. et al. CellProfiler 3.0: Next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).
pubmed: 29969450
pmcid: 6029841
doi: 10.1371/journal.pbio.2005970
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
pubmed: 22743772
doi: 10.1038/nmeth.2019
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).
pubmed: 33318659
doi: 10.1038/s41592-020-01018-x
Gómez-de-Mariscal, E. et al. DeepImageJ: a user-friendly plugin to run deep learning models in ImageJ. Preprint at http://biorxiv.org/lookup/doi/10.1101/799270 (2019).
Antun, V., Renna, F., Poon, C., Adcock, B. & Hansen, A. C. On instabilities of deep learning in image reconstruction and the potential costs of AI. Proc. Natl Acad. Sci. USA 117, 30088–30095 (2020).
Möckl, L., Roy, A. R. & Moerner, W. E. Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]. Biomed. Opt. Express 11, 1633 (2020).
pubmed: 32206433
pmcid: 7075610
doi: 10.1364/BOE.386361
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).
pubmed: 30224672
pmcid: 6212323
doi: 10.1038/s41592-018-0111-2
Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).
pubmed: 31133758
doi: 10.1038/s41592-019-0403-1
pmcid: 8759575
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. in Medical Image Computing and Computer-Assisted Intervention—MICCAI 2016 (eds. Ourselin, S. et al.) Vol. 9901, 424–432 (Springer International Publishing, 2016).
Nehme, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica 5, 458 (2018).
doi: 10.1364/OPTICA.5.000458
Isola, P., Zhu, J. Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017).
Zhu, J. Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017).
Bloice, M. D., Roth, P. M. & Holzinger, A. Biomedical image augmentation using Augmentor. Bioinformatics 35, 4522–4524 (2019).
pubmed: 30989173
doi: 10.1093/bioinformatics/btz259
Pan, S. J. & Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2010).
doi: 10.1109/TKDE.2009.191
Carneiro, T. et al. Performance analysis of google colaboratory as a tool for accelerating deep learning applications. IEEE Access 6, 61677–61685 (2018).
doi: 10.1109/ACCESS.2018.2874767
Bisong, E. Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners (Apress Imprint, Apress, 2019).
Melsted, P. et al. Modular and efficient pre-processing of single-cell RNA-seq. Preprint at http://biorxiv.org/lookup/doi/10.1101/673285 (2019).
Spiers, H. et al. Citizen science, cells and CNNs—deep learning for automatic segmentation of the nuclear envelope in electron microscopy data, trained with volunteer segmentations. Preprint at http://biorxiv.org/lookup/doi/10.1101/2020.07.28.223024 (2020).
Tinevez, J.-Y. et al. TrackMate: an open and extensible platform for single-particle tracking. Methods 115, 80–90 (2017).
pubmed: 27713081
doi: 10.1016/j.ymeth.2016.09.016
Quinn, J. A. et al. Deep convolutional neural networks for microscopy-based point of care diagnostics. in Machine Learning for Healthcare Conference (eds. Doshi-Velez, F. et al.) 271–281 (PMLR, 2016).
Tosheva, K. L., Yuan, Y., Matos Pereira, P., Culley, S. & Henriques, R. Between life and death: strategies to reduce phototoxicity in super-resolution microscopy. J. Phys. Appl. Phys. 53, 163001 (2020).
doi: 10.1088/1361-6463/ab6b95
Strack, R. Hessian structured illumination microscopy. Nat. Methods 15, 407–407 (2018).
pubmed: 29855581
doi: 10.1038/s41592-018-0023-1
Jin, L. et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun. 11, 1934 (2020).
pubmed: 32321916
pmcid: 7176720
doi: 10.1038/s41467-020-15784-x
Jacquemet, G., Carisey, A. F., Hamidi, H., Henriques, R. & Leterrier, C. The cell biologist’s guide to super-resolution microscopy. J. Cell Sci. 133, jcs240713 (2020).
Nelson, A. J. & Hess, S. T. Molecular imaging with neural training of identification algorithm (neural network localization identification). Microsc. Res. Tech. 81, 966–972 (2018).
pubmed: 30242941
pmcid: 6240359
doi: 10.1002/jemt.23059
Manor, U. et al. Deep learning‐based point‐scanning super‐resolution imaging. FASEB J. 34, 1–1 (2020).
doi: 10.1096/fasebj.2020.34.s1.02496
Owen, D. M. et al. PALM imaging and cluster analysis of protein heterogeneity at the cell surface. J. Biophotonics 3, 446–454 (2010).
pubmed: 20148419
doi: 10.1002/jbio.200900089
Sengupta, P. et al. Probing protein heterogeneity in the plasma membrane using PALM and pair correlation analysis. Nat. Methods 8, 969–975 (2011).
pubmed: 21926998
pmcid: 3400087
doi: 10.1038/nmeth.1704
Levet, F. et al. SR-Tesseler: a method to segment and quantify localization-based super-resolution microscopy data. Nat. Methods 12, 1065–1071 (2015).
pubmed: 26344046
doi: 10.1038/nmeth.3579
Heilemann, M. et al. Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes. Angew. Chem. Int. Ed. 47, 6172–6176 (2008).
doi: 10.1002/anie.200802376
Jungmann, R. et al. Multiplexed 3D cellular super-resolution imaging with DNA-PAINT and exchange-PAINT. Nat. Methods 11, 313–318 (2014).
pubmed: 24487583
pmcid: 4153392
doi: 10.1038/nmeth.2835
Ovesný, M., Křížek, P., Borkovec, J., Švindrych, Z. & Hagen, G. M. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).
pubmed: 24771516
pmcid: 4207427
doi: 10.1093/bioinformatics/btu202
Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).
pubmed: 29457791
pmcid: 5884429
doi: 10.1038/nmeth.4605
Goodfellow, I. et al. Generative adversarial nets. in Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z. et al.) 2672–2680 (Curran Associates, Inc., 2014).
Gustafsson, N. et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 (2016).
pubmed: 27514992
pmcid: 4990649
doi: 10.1038/ncomms12471
Laine, R. F. et al. NanoJ: a high-performance open-source super-resolution microscopy toolbox. J. Phys. Appl. Phys. 52, 163001 (2019).
doi: 10.1088/1361-6463/ab0261
Fazeli, E. et al. Automated cell tracking using StarDist and TrackMate. F1000Research 9, 1279 (2020).
pubmed: 33224481
pmcid: 7670479
doi: 10.12688/f1000research.27019.1
Rueden, C. T. et al. ImageJ2: ImageJ for the next generation of scientific image data. BMC Bioinforma. 18, 529 (2017).
doi: 10.1186/s12859-017-1934-z
LaChance, J. & Cohen, D. J. Practical fluorescence reconstruction microscopy for large samples and low-magnification imaging. PLoS Comput Biol. 16, e1008443 (2020).
pubmed: 33362219
pmcid: 7802935
doi: 10.1371/journal.pcbi.1008443
Moen, E. et al. Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. Preprint at http://biorxiv.org/lookup/doi/10.1101/803205 (2019).
von Chamier, L., Laine, R. F. & Henriques, R. Artificial intelligence for microscopy: what you should know. Biochem. Soc. Trans. 47, 1029–1040 (2019).
doi: 10.1042/BST20180391
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).
pubmed: 31285623
doi: 10.1038/s41592-019-0458-z
Nichols, J. A., Herbert Chan, H. W. & Baker, M. A. B. Machine learning: applications of artificial intelligence to imaging and diagnosis. Biophys. Rev. 11, 111–118 (2019).
pubmed: 30182201
doi: 10.1007/s12551-018-0449-9
Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
pubmed: 15376593
doi: 10.1109/TIP.2003.819861
Kirillov, A., He, K., Girshick, R., Rother, C. & Dollár, P. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9404–9413 (2019).
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. & Zisserman, A. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2010).
doi: 10.1007/s11263-009-0275-4
Everingham, M. & Winn, J. The Pascal Visual Object Classes Challenge 2012 (VOC2012) Development Kit. Pattern Analysis, Statistical Modelling and Computational Learning, Technical Report. vol. 8 (VOC2012, 2011).
Lavoie-Cardinal, F. et al. MICRA-Net: MICRoscopy Analysis Neural Network to solve detection, classification, and segmentation from a single simple auxiliary task. https://www.researchsquare.com/article/rs-95613/v1 (2020).
Lavoie-Cardinal, F. et al. Neuronal activity remodels the F-actin based submembrane lattice in dendrites but not axons of hippocampal neurons. Sci. Rep. 10, 11960 (2020).
pubmed: 32686703
pmcid: 7371643
doi: 10.1038/s41598-020-68180-2
Hollandi, R., Diósdi, Á., Hollandi, G., Moshkov, N. & Horváth, P. AnnotatorJ: an ImageJ plugin to ease hand annotation of cellular compartments. Mol. Biol. Cell 31, 2179–2186 (2020).
pubmed: 32697683
pmcid: 7550707
doi: 10.1091/mbc.E20-02-0156
Speiser, A. et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Preprint at http://biorxiv.org/lookup/doi/10.1101/2020.10.26.355164 (2020).
Khadangi, A., Boudier, T. & Rajagopal, V. EM-stellar: benchmarking deep learning for electron microscopy image segmentation. Bioinformatics. https://doi.org/10.1093/bioinformatics/btaa1094 , btaa1094 (2021).
Stubb, A. et al. Fluctuation-based super-resolution traction force microscopy. Nano Lett. 20, 2230–2245 (2020).
pubmed: 32142297
pmcid: 7146861
doi: 10.1021/acs.nanolett.9b04083
Jacquemet, G. et al. FiloQuant reveals increased filopodia density during breast cancer progression. J. Cell Biol. 216, 3387–3403 (2017).
pubmed: 28765364
pmcid: 5626550
doi: 10.1083/jcb.201704045
Legant, W. R. et al. High-density three-dimensional localization microscopy across large volumes. Nat. Methods 13, 359–365 (2016).
pubmed: 26950745
pmcid: 4889433
doi: 10.1038/nmeth.3797
Vassilopoulos, S., Gibaud, S., Jimenez, A., Caillol, G. & Leterrier, C. Ultrastructure of the axonal periodic scaffold reveals a braid-like organization of actin rings. Nat. Commun. 10, 5803 (2019).
pubmed: 31862971
pmcid: 6925202
doi: 10.1038/s41467-019-13835-6
Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020).
pubmed: 32939066
pmcid: 7759461
doi: 10.1038/s41586-020-2649-2
Martín, A. et al. TensorFlow: large-scale machine learning on heterogeneous systems. Preprint at https://arxiv.org/abs/1603.04467 (2015).
Arganda-Carreras, I. et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Front. Neuroanat. 9, 142 (2015).
Cardona, A. et al. An integrated micro- and macroarchitectural analysis of the Drosophila brain by computer-assisted serial section electron microscopy. PLoS Biol. 8, e1000502 (2010).
pubmed: 20957184
pmcid: 2950124
doi: 10.1371/journal.pbio.1000502
Wortel, I. M. N., Dannenberg, K., Berry, J. C., Miller, M. J. & Textor, J. CelltrackR: an R package for fast and flexible analysis of immune cell migration data. Preprint at http://biorxiv.org/lookup/doi/10.1101/670505 (2019).
Kaukonen, R., Jacquemet, G., Hamidi, H. & Ivaska, J. Cell-derived matrices for studying cell proliferation and directional migration in a complex 3D microenvironment. Nat. Protoc. 12, 2376–2390 (2017).
pubmed: 29048422
doi: 10.1038/nprot.2017.107
Jimenez, A., Friedl, K. & Leterrier, C. About samples, giving examples: optimized single molecule localization microscopy. Methods 174, 100–114 (2020).
pubmed: 31078795
doi: 10.1016/j.ymeth.2019.05.008
Mlodzianoski, M. J. et al. Sample drift correction in 3D fluorescence photoactivation localization microscopy. Opt. Express 19, 15009 (2011).
pubmed: 21934862
doi: 10.1364/OE.19.015009
Jacquemet, G. et al. L-type calcium channels regulate filopodia stability and cancer cell invasion downstream of integrin signalling. Nat. Commun. 7, 13297 (2016).
pubmed: 27910855
pmcid: 5146291
doi: 10.1038/ncomms13297
Jacquemet, G. ZeroCostDL4Mic—CARE (3D) example training and test dataset. https://doi.org/10.5281/ZENODO.3713337 (2020).
Jacquemet, G. ZeroCostDL4Mic—CARE (2D) example training and test dataset. https://doi.org/10.5281/ZENODO.3713330 (2020).
Jacquemet, G. ZeroCostDL4Mic—Noise2Void (3D) example training and test dataset. https://doi.org/10.5281/ZENODO.3713326 (2020).
Stubb, A., Jacquemet, G. & Ivaska, J. ZeroCostDL4Mic—Noise2Void (2D) example training and test dataset. https://doi.org/10.5281/ZENODO.3713315 (2020).
Leterrier, C. & Laine, R. F. ZeroCostDL4Mic—DeepSTORM training and example dataset. https://doi.org/10.5281/ZENODO.3959089 (2020).
Jacquemet, G. ZeroCostDL4Mic—CycleGAN example training and test dataset. https://doi.org/10.5281/ZENODO.3941884 (2020).
Jacquemet, G. ZeroCostDL4Mic—pix2pix example training and test dataset. https://doi.org/10.5281/ZENODO.3941889 (2020).
Jacquemet, G. & Chamier, L. V. ZeroCostDL4Mic—YoloV2 example training and test dataset. https://doi.org/10.5281/ZENODO.3941908 (2020).
Jukkala, J. & Jacquemet, G. ZeroCostDL4Mic—Stardist example training and test dataset. https://doi.org/10.5281/ZENODO.3715492 (2020).
Spahn, C. & Heilemann, M. ZeroCostDL4Mic—label-free prediction (fnet) example training and test dataset. https://doi.org/10.5281/ZENODO.3748967 (2020).
Laine, R. F. et al. HenriquesLab/ZeroCostDL4Mic: 1.12.2. Zenodo https://doi.org/10.5281/ZENODO.4543673 (2021).
Postma, M. & Goedhart, J. PlotsOfData—a web app for visualizing data together with their summaries. PLoS Biol. 17, e3000202 (2019).
pubmed: 30917112
pmcid: 6453475
doi: 10.1371/journal.pbio.3000202