Deep learning for cellular image analysis.
Journal
Nature methods
ISSN: 1548-7105
Titre abrégé: Nat Methods
Pays: United States
ID NLM: 101215604
Informations de publication
Date de publication:
12 2019
12 2019
Historique:
received:
24
10
2018
accepted:
03
04
2019
pubmed:
28
5
2019
medline:
14
2
2020
entrez:
29
5
2019
Statut:
ppublish
Résumé
Recent advances in computer vision and machine learning underpin a collection of algorithms with an impressive ability to decipher the content of images. These deep learning algorithms are being applied to biological images and are transforming the analysis and interpretation of imaging data. These advances are positioned to render difficult analyses routine and to enable researchers to carry out new, previously impossible experiments. Here we review the intersection between deep learning and cellular image analysis and provide an overview of both the mathematical mechanics and the programming frameworks of deep learning that are pertinent to life scientists. We survey the field's progress in four key applications: image classification, image segmentation, object tracking, and augmented microscopy. Last, we relay our labs' experience with three key aspects of implementing deep learning in the laboratory: annotating training data, selecting and training a range of neural network architectures, and deploying solutions. We also highlight existing datasets and implementations for each surveyed application.
Identifiants
pubmed: 31133758
doi: 10.1038/s41592-019-0403-1
pii: 10.1038/s41592-019-0403-1
pmc: PMC8759575
mid: NIHMS1744527
doi:
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
1233-1246Subventions
Organisme : NCI NIH HHS
ID : U24 CA224309
Pays : United States
Références
Grimm, J. B. et al. A general method to fine-tune fluorophores for live-cell and in vivo imaging. Nat. Methods 14, 987–994 (2017).
pubmed: 28869757
pmcid: 5621985
Liu, H. et al. Visualizing long-term single-molecule dynamics in vivo by stochastic protein labeling. Proc. Natl. Acad. Sci. USA 115, 343–348 (2018).
pubmed: 29284749
Regot, S., Hughey, J. J., Bajar, B. T., Carrasco, S. & Covert, M. W. High-sensitivity measurements of multiple kinase activities in live single cells. Cell 157, 1724–1734 (2014).
pubmed: 24949979
pmcid: 4097317
Sampattavanich, S. et al. Encoding growth factor identity in the temporal dynamics of FOXO3 under the combinatorial control of ERK and AKT kinases. Cell Syst. 6, 664–678 (2018).
pubmed: 29886111
pmcid: 6322215
Megason, S. G. In toto imaging of embryogenesis with confocal time-lapse microscopy. Methods Mol. Biol. 546, 317–332 (2009).
pubmed: 19378112
pmcid: 2826616
Udan, R. S., Piazza, V. G., Hsu, C. W., Hadjantonakis, A.-K. & Dickinson, M. E. Quantitative imaging of cell dynamics in mouse embryos using light-sheet microscopy. Development 141, 4406–4414 (2014).
pubmed: 25344073
pmcid: 4302910
Chen, B.-C. et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).
pubmed: 4336192
pmcid: 4336192
Royer, L. A. et al. Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms. Nat. Biotechnol. 34, 1267–1278 (2016).
pubmed: 27798562
pmcid: 27798562
McDole, K. et al. In toto imaging and reconstruction of post-implantation mouse development at the single-cell level. Cell 175, 859–876 (2018).
pubmed: 30318151
Shah, S., Lubeck, E., Zhou, W. & Cai, L. seqFISH accurately detects transcripts in single cells and reveals robust spatial organization in the hippocampus. Neuron 94, 752–758 (2017).
pubmed: 28521130
Keren, L. et al. A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging. Cell 174, 1373–1387 (2018).
pubmed: 30193111
pmcid: 6132072
Lin, J.-R. et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. eLife 7, e31657 (2018).
pubmed: 29993362
pmcid: 6075866
Caicedo, J. C. et al. Data-analysis strategies for image-based cell profiling. Nat. Methods 14, 849–863 (2017).
pubmed: 28858338
pmcid: 6871000
van der Walt, S., Colbert, S. C. & Varoquaux, G. The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13, 22–30 (2011).
Jones, E. et al. SciPy: open source scientific tools for Python. https://www.scipy.org/ (2001).
McKinney, W. Data structures for statistical computing in Python. In Proc. 9th Python in Science Conference (eds. van der Walt, S. & Millman, J.) 51–56 (SciPy, 2010).
van der Walt, S. et al. scikit-image: image processing in Python. PeerJ 2, e453 (2014).
pubmed: 25024921
pmcid: 25024921
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
Kluyver, T. et al. Jupyter Notebooks—a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas (eds. Loizides, F. & Schmidt, B.) 87–90 (IOS Press, 2016).
Stylianidou, S., Brennan, C., Nissen, S. B., Kuwada, N. J. & Wiggins, P. A. SuperSegger: robust image segmentation, analysis and lineage tracking of bacterial cells. Mol. Microbiol. 102, 690–700 (2016).
pubmed: 27569113
Paintdakhi, A. et al. Oufti: an integrated software package for high-accuracy, high-throughput quantitative microscopy analysis. Mol. Microbiol. 99, 767–777 (2016).
pubmed: 26538279
Ursell, T. et al. Rapid, precise quantification of bacterial cellular dimensions across a genomic-scale knockout library. BMC Biol. 15, 17 (2017).
pubmed: 28222723
pmcid: 5320674
Carpenter, A. E. et al. CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100 (2006).
pubmed: 17076895
pmcid: 17076895
McQuin, C. et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).
pubmed: 29969450
pmcid: 29969450
Sommer, C., Straehle, C., Köthe, U. & Hamprecht, F. A. Ilastik: interactive learning and segmentation toolkit. In 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro (Wright, S., Pan, X. & Liebling, M.) 230–233 (IEEE, 2011).
Belevich, I., Joensuu, M., Kumar, D., Vihinen, H. & Jokitalo, E. Microscopy Image Browser: a platform for segmentation and analysis of multidimensional datasets. PLoS Biol. 14, e1002340 (2016).
pubmed: 26727152
pmcid: 4699692
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
pubmed: 22743772
pmcid: 22743772
Allan, C. et al. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods 9, 245–253 (2012).
pubmed: 22373911
pmcid: 3437820
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
pubmed: 26017442
pmcid: 26017442
Krizhevsky, A., Sutskever, I. & Hinton, G.E. ImageNet classification with deep convolutional neural networks. In Proc. 25th International Conference on Neural Information Processing Systems (eds. Pereira, F. et al.) 1090–1098 (Curran Associates, 2012).
Carrasquilla, J. & Melko, R. G. Machine learning phases of matter. Nat. Phys. 13, 431–434 (2017).
Nguyen, T. Q. et al. Topology classification with deep learning to improve real-time event selection at the LHC. Preprint available at https://arxiv.org/abs/1807.00083 (2018).
Castelvecchi, D. Artificial intelligence called in to tackle LHC data deluge. Nature 528, 18–19 (2015).
pubmed: 26632567
Ramsundar, B. et al. Massively multitask networks for drug discovery. Preprint available at http://arxiv.org/abs/1502.02072 (2015).
Feinberg, E. N. et al. Spatial graph convolutions for drug discovery. Preprint available at http://arxiv.org/abs/1803.04465 (2018).
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
pubmed: 30224757
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Poplin, R. et al. A universal SNP and small-indel variant caller using deep neural networks. Nat. Biotechnol. 36, 983–987 (2018).
pubmed: 30247488
Zhou, J. et al. Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk. Nat. Genet. 50, 1171–1179 (2018).
pubmed: 30013180
pmcid: 6094955
Alipanahi, B., Delong, A., Weirauch, M. T. & Frey, B. J. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33, 831–838 (2015).
Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Syst. Biol. 12, 878 (2016).
pubmed: 27474269
pmcid: 4965871
Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).
Karpathy, A. Software 2.0. Medium https://medium.com/@karpathy/software-2-0-a64152b37c35 (2017).
Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).
pubmed: 28778026
Xing, F., Xie, F., Su, H., Liu, F. & Yang, L. Deep learning in microscopy image analysis: a survey. IEEE Trans. Neural Netw. Learn. Syst. 29, 4550–4568 (2018).
Smith, K. et al. Phenotypic image analysis software tools for exploring and understanding big image data from cell-based assays. Cell Syst. 6, 636–653 (2018).
pubmed: 29953863
Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLOS Comput. Biol. 12, e1005177 (2016).
pubmed: 27814364
pmcid: 5096676
Cireşan, D. C., Meier, U., Gambardella, L. M. & Schmidhuber, J. Deep, big, simple neural nets for handwritten digit recognition. Neural Comput. 22, 3207–3220 (2010).
pubmed: 20858131
Zhang, W. et al. Deep model based transfer and multi-task learning for biological image analysis. IEEE Trans. Big Data https://doi.org/10.1109/TBDATA.2016.2573280 (2016).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Proc. 27th International Conference on Neural Information Processing Systems (eds. Ghahramani, Z. et al.) 3320–3328 (MIT Press, 2014).
Caicedo, J.C. et al. Evaluation of deep learning strategies for nucleus segmentation in fluorescence images. Preprint available at https://www.biorxiv.org/content/early/2018/06/16/335216 (2018).
Newby, J. M., Schaefer, A. M., Lee, P. T., Forest, M. G. & Lai, S. K. Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D. Proc. Natl. Acad. Sci. USA 115, 9026–9031 (2018).
pubmed: 30135100
Sadanandan, S. K., Ranefall, P., Le Guyader, S. & Wählby, C. Automated training of deep convolutional neural networks for cell segmentation. Sci. Rep. 7, 7860 (2017).
pubmed: 28798336
pmcid: 28798336
Chen, J. et al. The Allen Cell Structure Segmenter: a new open source toolkit for segmenting 3D intracellular structures in fluorescence microscopy images. Preprint available at https://www.biorxiv.org/content/early/2018/12/08/491035 (2018).
Hughes, A. J. et al. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat. Methods 15, 587–590 (2018).
pubmed: 30065368
pmcid: 30065368
Sullivan, D. P. et al. Deep learning is combined with massive-scale citizen science to improve large-scale image classification. Nat. Biotechnol. 36, 820–828 (2018).
pubmed: 30125267
pmcid: 30125267
Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In Proc. 12th USENIX Conference on Operating Systems Design and Implementation (eds. Keeton, K. & Roscoe, T.) 265–283 (USENIX Association, 2016).
Chollet, F. Keras. GitHub https://github.com/keras-team/keras (2015).
Paszke, A. et al. Automatic differentiation in PyTorch. Oral presentation at NIPS 2017 Workshop on Automatic Differentiation, Long Beach, CA, USA, 9 December 2017.
Chen, T. et al. MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. Preprint available at http://arxiv.org/abs/1512.01274 (2015).
Seide, F. & Agarwal, A. CNTK: Microsoft’s open-source deep-learning toolkit. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (eds. Krishnapuram, B. et al.) 2135 (ACM, 2016).
Bergstra, J. et al. Theano: deep learning on GPUs with Python. Paper presented at Big Learning 2011: NIPS 2011 Workshop on Algorithms, Systems, and Tools for Learning at Scale, Sierra Nevada, Spain, 16–17 December 2011.
Jia, Y. et al. Caffe: convolutional architecture for fast feature embedding. In Proc. 22nd ACM International Conference on Multimedia (eds. Hua, K. A. et al.) 675–678 (ACM, 2014).
Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. In Proc. 44th Annual International Symposium on Computer Architecture (eds. Moshovos, A. et al.) 1–12 (ACM, 2017).
Owens, J. D. et al. GPU computing. Proc. IEEE 96, 879–899 (2008).
Chetlur, S. et al. cuDNN: efficient primitives for deep learning. Preprint available at http://arxiv.org/abs/1410.0759 (2014).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. 29th IEEE Conference on Computer Vision and Pattern Recognition (eds. Agapito, L. et al.) 770–778 (IEEE, 2016).
Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (eds. Liu, Y. et al.) 2261–2269 (IEEE, 2017).
Pelt, D. M. & Sethian, J. A. A mixed-scale dense convolutional neural network for image analysis. Proc. Natl. Acad. Sci. USA 115, 254–259 (2018).
pubmed: 29279403
Bishop, C. M. Pattern Recognition and Machine Learning (Information Science and Statistics) (Springer-Verlag, 2006).
Ebrahimi, M. S. & Abadi, H. K. Study of residual networks for image recognition. Preprint available at http://arxiv.org/abs/1805.00325 (2018).
Richardson, L. & Ruby, S. RESTful Web Services (O’Reilly Media, 2007).
Merkel, D. Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014, 2 (2014).
Haberl, M. G. et al. CDeep3M-Plug-and-Play cloud-based deep learning for image segmentation. Nat. Methods 15, 677–680 (2018).
pubmed: 30171236
pmcid: 6548193
Pawlowski, N., Caicedo, J. C., Singh, S., Carpenter, A. E. & Storkey, A. Automating morphological profiling with generic deep convolutional networks. Preprint available at https://www.biorxiv.org/content/early/2016/11/02/085118 (2016).
Godinez, W. J., Hossain, I., Lazic, S. E., Davies, J. W. & Zhang, X. A multi-scale convolutional neural network for phenotyping high-content cellular images. Bioinformatics 33, 2010–2019 (2017).
pubmed: 28203779
Kandaswamy, C., Silva, L. M., Alexandre, L. A. & Santos, J. M. High-content analysis of breast cancer using single-cell deep transfer learning. J. Biomol. Screen. 21, 252–259 (2016).
pubmed: 26746583
Sommer, C., Hoefler, R., Samwer, M. & Gerlich, D. W. A deep learning and novelty detection framework for rapid phenotyping in high-content screening. Mol. Biol. Cell 28, 3428–3436 (2017).
pubmed: 28954863
pmcid: 28954863
Simm, J. et al. Repurposing high-throughput image assays enables biological activity prediction for drug discovery. Cell Chem. Biol. 25, 611–618 (2018).
pubmed: 29503208
pmcid: 6031326
Buggenthin, F. et al. Prospective identification of hematopoietic lineage choice by deep learning. Nat. Methods 14, 403–406 (2017).
pubmed: 28218899
pmcid: 5376497
Kraus, O. Z., Ba, J. L. & Frey, B. J. Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics 32, i52–i59 (2016).
pubmed: 4908336
pmcid: 4908336
Kraus, O. Z. et al. Automated analysis of high-content microscopy data with deep learning. Mol. Syst. Biol. 13, 924 (2017).
pubmed: 28420678
pmcid: 5408780
Pärnamaa, T. & Parts, L. Accurate classification of protein subcellular localization from high-throughput microscopy images using deep learning. G3 (Bethesda) 7, 1385–1392 (2017).
Nitta, N. et al. Intelligent image-activated cell sorting. Cell 175, 266–276 (2018).
pubmed: 30166209
Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015 (eds. Navab, N. et al.) 234–241 (Springer, 2015).
Bai, M. & Urtasun, R. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (eds. Liu, Y. et al.) 2858–2866 (IEEE, 2017).
Wang, W. et al. Learn to segment single cells with deep distance estimator and deep cell detector. Preprint available at https://arxiv.org/abs/1803.10829 (2018).
Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28 (eds. Cortes, C. et al.) 91–99 (Curran Associates, 2015).
Lin, T., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) (eds. Ikeuchi, K. et al.) 2999–3007 (IEEE, 2018).
He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV) (eds. Ikeuchi, K. et al.) 2980–2988 (IEEE, 2018).
Johnson, J. W. Adapting Mask-RCNN for automatic nucleus segmentation. Preprint available at http://arxiv.org/abs/1805.00500 (2018).
Tsai, H.-F., Gajda, J., Sloan, T. F. W., Rares, A. & Shen, A. Q. Usiigaci: instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning. Preprint available at https://www.biorxiv.org/content/early/2019/01/18/524041 (2019).
Hollandi, R. et al. A deep learning framework for nucleus segmentation using image style transfer. Preprint available at https://www.biorxiv.org/content/10.1101/580605v1 (2019).
De Brabandere, B., Neven, D. & Van Gool, L. Semantic instance segmentation with a discriminative loss function. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Liu, Y. et al.) 478–480 (IEEE, 2017).
Payer, C., Štern, D., Neff, T., Bischof, H. & Urschler, M. Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018 (eds. Frangi, A. F. et al.) 3–11 (Springer, 2018).
Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. Preprint available at http://arxiv.org/abs/1703.10593 (2017).
Haering, M., Grosshans, J., Wolf, F. & Eule, S. Automated segmentation of epithelial tissue using cycle-consistent generative adversarial networks. Preprint available at https://www.biorxiv.org/content/early/2018/04/30/311373 (2018).
Mahmood, F. et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. Preprint available at http://arxiv.org/abs/1810.00236 (2018).
Tokuoka, Y. et al. Convolutional neural network-based instance segmentation algorithm to acquire quantitative criteria of early mouse development. Preprint available at https://www.biorxiv.org/content/early/2018/06/01/324186 (2018).
Januszewski, M. et al. High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 15, 605–610 (2018).
pubmed: 30013046
Li, P. H. et al. Automated reconstruction of a serial-section EM Drosophila brain with flood-filling networks and local realignment. Preprint at https://www.biorxiv.org/content/10.1101/605634v1 (2019).
Booz Allen Hamilton. 2018 Data Science Bowl. Kaggle https://www.kaggle.com/c/data-science-bowl-2018 (2018).
Facchetti, G., Knapp, B., Flor-Parra, I., Chang, F. & Howard, M. Reprogramming Cdr2-dependent geometry-based cell size control in fission yeast. Curr. Biol. 29, 350–358 (2019).
pubmed: 30639107
pmcid: 6345630
Khoshdeli, M., Winkelmaier, G. & Parvin, B. Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes. BMC Bioinforma. 19, 294 (2018).
Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imaging 36, 1550–1560 (2017).
pubmed: 28287963
Regev, A. et al. The Human Cell Atlas. eLife 6, e27041 (2017).
pubmed: 29206104
pmcid: 5762154
Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A. & Teichmann, S. A. The Human Cell Atlas: from vision to reality. Nature 550, 451–453 (2017).
pubmed: 29072289
Purvis, J. E. & Lahav, G. Encoding and decoding cellular information through signaling dynamics. Cell 152, 945–956 (2013).
pubmed: 23452846
pmcid: 3707615
Kimmel, J. C., Chang, A. Y., Brack, A. S. & Marshall, W. F. Inferring cell state by quantitative motility analysis reveals a dynamic state system and broken detailed balance. PLoS Comput. Biol. 14, e1005927 (2018).
pubmed: 29338005
pmcid: 5786322
Wang, P. et al. Robust growth of Escherichia coli. Curr. Biol. 20, 1099–1103 (2010).
pubmed: 20537537
pmcid: 2902570
Dow, J. A., Lackie, J. M. & Crocket, K. V. A simple microcomputer-based system for real-time analysis of cell behaviour. J. Cell Sci. 87, 171–182 (1987).
pubmed: 3667712
Levine, M. D., Youssef, Y. M., Noble, P. B. & Boyarsky, A. The quantification of blood cell motion by a method of automatic digital picture processing. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-2, 444–450 (1980).
Smal, I., Niessen, W. & Meijering, E. Bayesian tracking for fluorescence microscopic imaging. In 3rd IEEE International Symposium on Biomedical Imaging: Macro to Nano, 2006 (eds. Kovačević, J. et al.) 550–553 (IEEE, 2006).
Godinez, W. et al. Tracking of virus particles in time-lapse fluorescence microscopy image sequences. In 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro (eds. Fessler, J. et al.) 256–259 (IEEE, 2007).
Ngoc, S. N., Briquet-Laugier, F., Boulin, C. & Olivo, J.-C. Adaptive detection for tracking moving biological objects in video microscopy sequences. In Proc. International Conference on Image Processing (eds. Chang, S.-F. et al.) 484–487 (IEEE, 1997).
Kachouie, N. N. & Fieguth, P. W. Extended-Hungarian-JPDA: exact single-frame stem cell tracking. IEEE Trans. Biomed. Eng. 54, 2011–2019 (2007).
pubmed: 18018696
Meijering, E., Dzyubachyk, O., Smal, I. & van Cappellen, W. A. Tracking in cell and developmental biology. Semin. Cell Dev. Biol. 20, 894–902 (2009).
pubmed: 19660567
Jaqaman, K. et al. Robust single-particle tracking in live-cell time-lapse sequences. Nat. Methods 5, 695–702 (2008).
pubmed: 18641657
pmcid: 2747604
Tinevez, J.-Y. et al. TrackMate: an open and extensible platform for single-particle tracking. Methods 115, 80–90 (2017).
Cooper, S., Barr, A. R., Glen, R. & Bakal, C. NucliTrack: an integrated nuclei tracking application. Bioinformatics 33, 3320–3322 (2017).
pubmed: 28637183
pmcid: 5860035
Magnusson, K. E. G., Jalden, J., Gilbert, P. M. & Blau, H. M. Global linking of cell tracks using the Viterbi algorithm. IEEE Trans. Med. Imaging 34, 911–929 (2015).
pubmed: 25415983
Amat, F. et al. Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nat. Methods 11, 951–958 (2014).
pubmed: 25042785
Akram, S. U., Kannala, J., Eklund, L. & Heikkilä, J. Cell tracking via proposal generation and selection. Preprint available at https://arxiv.org/abs/1705.03386 (2017).
Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 (eds. Mori, K. et al.) 411–418 (Springer, 2013).
Nie, W.-Z., Li, W.-H., Liu, A.-A., Hao, T. & Su, Y.-T. 3D convolutional networks-based mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Agapito, L. et al.) 55–62 (IEEE, 2016).
Mao, Y. & Yin, Z. A hierarchical convolutional neural network for mitosis detection in phase-contrast microscopy images. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2016 (eds. Ourselin, S. et al.) 685–692 (Springer, 2016).
Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).
pubmed: 30127430
Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2019).
pubmed: 30573820
pmcid: 30573820
Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C., Heras, F. J. H. & de Polavieja, G. G. idtracker.ai: tracking all individuals in small or large collectives of unmarked animals. Nat. Methods 16, 179–182 (2019).
pubmed: 30643215
Gordon, D., Farhadi, A. & Fox, D. Re
Cui, Z., Xiao, S., Feng, J. & Yan, S. Recurrently target-attending tracking. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (eds. Agapito, L. et al.) 1449–1458 (IEEE, 2016).
Wang, Y., Mao, H. & Yi, Z. Stem cell motion-tracking by using deep neural networks with multi-output. Neural Comput. Appl. https://doi.org/10.1007/s00521-017-3291-2 (2017).
doi: 10.1007/s00521-017-3291-2
pubmed: 29051688
Sadeghian, A., Alahi, A. & Savarese, S. Tracking the untrackable: learning to track multiple cues with long-term dependencies. In 2017 IEEE International Conference on Computer Vision (eds. Ikeuchi, K. et al.) 300–311 (IEEE, 2017).
Zhang, D., Maei, H., Wang, X. & Wang, Y.-F. Deep reinforcement learning for visual object tracking in videos. Preprint available at http://arxiv.org/abs/1701.08936 (2017).
Wen, C. et al. Deep-learning-based flexible pipeline for segmenting and tracking cells in 3D image time series for whole brain imaging. Preprint available at https://www.biorxiv.org/content/early/2018/08/06/385567 (2018).
Sullivan, D. P. & Lundberg, E. Seeing more: a future of augmented microscopy. Cell 173, 546–548 (2018).
pubmed: 29677507
Ounkomol, C. et al. Three dimensional cross-modal image inference: label-free methods for subcellular structure prediction. Preprint available at https://www.biorxiv.org/content/10.1101/216606v4 (2017).
Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).
pubmed: 29656897
pmcid: 29656897
Johnson, G. R., Donovan-Maiye, R. M. & Maleckar, M. M. Building a 3D integrated cell. Preprint available at https://www.biorxiv.org/content/early/2017/12/21/238378 (2017).
Osokin, A., Chessel, A., Salas, R. E. C. & Vaggi, F. GANs for biological image synthesis. In 2017 IEEE International Conference on Computer Vision (eds. Ikeuchi, K. et al.) 2252–2261 (IEEE, 2017).
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).
pubmed: 30224672
pmcid: 30224672
Johnson, G., Donovan-Maiye, R., Ounkomol, C. & Maleckar, M. M. Studying stem cell organization using “label-free” methods and a novel generative adversarial model. Biophys. J. 114, 43a (2018).
Stumpe, M. & Mermel, C. An augmented reality microscope for cancer detection. Google AI Blog https://ai.googleblog.com/2018/04/an-augmented-reality-microscope.html (2018).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Preprint available at https://www.preprints.org/manuscript/201812.0137/v1 (2018).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Preprint available at https://www.biorxiv.org/content/early/2018/07/03/236463 (2018).
Wang, H. et al. Deep learning achieves super-resolution in fluorescence microscopy. Preprint available at https://www.biorxiv.org/content/early/2018/04/27/309641 (2018).
Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).
Angelo, M. et al. Multiplexed ion beam imaging of human breast tumors. Nat. Med. 20, 436–442 (2014).
pubmed: 24584119
pmcid: 4110905
Acuna, D., Ling, H., Kar, A. & Fidler, S. Efficient interactive annotation of segmentation datasets with Polygon-RNN++. Preprint available at http://arxiv.org/abs/1803.09693 (2018).
Zoph, B. & Le, Q.V. Neural architecture search with reinforcement learning. Preprint available at http://arxiv.org/abs/1611.01578 (2016).
Zoph, B., Vasudevan, V., Shlens, J. & Le, Q.V. Learning transferable architectures for scalable image recognition. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (eds. Forsyth, D. et al.) 8697–8710 (IEEE, 2018).
Jackson, A. S., Bulat, A., Argyriou, V. & Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. Preprint available at http://arxiv.org/abs/1703.07834 (2017).
Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Proc. 27th International Conference on Machine Learning (eds. Fürnkranz, J. & Joachims, T.) 807–814 (Omnipress, 2010).
Li, H., Zhao, R. & Wang, X. Highly efficient forward and backward propagation of convolutional neural networks for pixelwise classification. Preprint available at http://arxiv.org/abs/1412.4526 (2014).
Chollet, F. Xception: deep learning with depthwise separable convolutions. In Proc. 30th IEEE Conference on Computer Vision and Pattern Recognition (eds. Liu, Y. et al.) 1800–1807 (IEEE, 2017).
Howard, A. G. et al. MobileNets: efficient convolutional neural networks for mobile vision applications. Preprint available at https://arxiv.org/abs/1704.04861v1 (2017).
Lin, T. et al. Feature pyramid networks for object detection. In Proc. 30th IEEE Conference on Computer Vision and Pattern Recognition (eds. Liu, Y. et al.) 936–944 (IEEE, 2017).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Preprint available at http://arxiv.org/abs/1502.03167 (2015).
Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. How does batch normalization help optimization? (No, it is not about internal covariate shift). Preprint available at http://arxiv.org/abs/1805.11604 (2018).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
Li, X., Chen, S., Hu, X. & Yang, J. Understanding the disharmony between dropout and batch normalization by variance shift. Preprint available at http://arxiv.org/abs/1801.05134 (2018).
Bannon, D. et al. DeepCell 2.0: automated cloud deployment of deep learning models for large-scale cellular image analysis. Preprint available at https://www.biorxiv.org/content/early/2018/12/22/505032 (2018).
Thul, P. J. et al. A subcellular map of the human proteome. Science 356, eaal3321 (2017).
Ljosa, V., Sokolnicki, K. L. & Carpenter, A. E. Annotated high-throughput microscopy image sets for validation. Nat. Methods 9, 637 (2012).
pubmed: 22743765
pmcid: 22743765
Maška, M. et al. A benchmark for comparison of cell tracking algorithms. Bioinformatics 30, 1609–1617 (2014).
pubmed: 24526711
pmcid: 4029039
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In Proc. 2015 IEEE International Conference on Computer Vision (eds. Bajcsy, R. et al.) 1026–1034 (IEEE, 2015).
Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964).
Nesterov, Y. E. A method for solving the convex programming problem with convergence rate O (1/k
Sutskever, I., Martens, J., Dahl, G. & Hinton, G. On the importance of initialization and momentum in deep learning. Proc. Mach. Learn. Res. 28, 1139–1147 (2013).
Tieleman, T. & Hinton, G. Neural Networks for Machine Learning lecture 6.5—rmsprop: divide the gradient by a running average of its recent magnitude. Coursera https://www.coursera.org/learn/neural-networks (2012).
Duchi, J., Hazan, E. & Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011).
Zeiler, M. D. ADADELTA: an adaptive learning rate method. Preprint available at http://arxiv.org/abs/1212.5701 (2012).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint available at http://arxiv.org/abs/1412.6980 (2014).
Wilson, A. C., Roelofs, R., Stern, M., Srebro, N. & Recht, B. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30 (eds. Guyon, I. et al.) 4148–4158 (Curran Associates, Inc., 2017).
Keskar, N. S. & Socher, R. Improving generalization performance by switching from Adam to SGD. Preprint available at http://arxiv.org/abs/1712.07628 (2017).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Sjoberg, J. & Ljung, L. Overtraining, regularization and searching for a minimum, with application to neural networks. Int. J. Control 62, 1391–1407 (1995).
Ting, K. M. Confusion matrix. In Encyclopedia of Machine Learning and Data Mining (eds. Sammut, C. & Webb, G. I.) 260–260 (Springer, 2017).
Bajcsy, P. et al. Survey statistics of automated segmentations applied to optical imaging of mammalian cells. BMC Bioinforma. 16, 330 (2015).
Sokolova, M. & Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45, 427–437 (2009).
Everingham, M., Van Gool, L., Williams, C. K., Winn, J. & Zisserman, A. The Pascal Visual Object Classes (voc) challenge. Int. J. Comput. Vis. 88, 303–338 (2010).
Kotila, M. Hyperparameter Optimization for Keras Models (Autonomio, 2018).