Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.
Journal
Nature methods
ISSN: 1548-7105
Titre abrégé: Nat Methods
Pays: United States
ID NLM: 101215604
Informations de publication
Date de publication:
25 Jun 2024
25 Jun 2024
Historique:
received:
01
05
2023
accepted:
17
05
2024
medline:
26
6
2024
pubmed:
26
6
2024
entrez:
25
6
2024
Statut:
aheadofprint
Résumé
Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.
Identifiants
pubmed: 38918605
doi: 10.1038/s41592-024-02319-1
pii: 10.1038/s41592-024-02319-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : Gatsby Charitable Foundation
ID : GAT3708
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : National Science Foundation (NSF)
ID : IOS-2115007
Organisme : National Science Foundation (NSF)
ID : 1707398
Organisme : Simons Foundation
ID : 543023
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (NINDS)
ID : U19NS123716
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : Wellcome Trust
ID : 216324
Pays : United Kingdom
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (NINDS)
ID : K99NS128075
Organisme : Nederlandse Organisatie voor Wetenschappelijk Onderzoek (Netherlands Organisation for Scientific Research)
ID : VI.Veni.212.184
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (NINDS)
ID : NS075023
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Diabetes and Digestive and Kidney Diseases (National Institute of Diabetes & Digestive & Kidney Diseases)
ID : 5R01DK131086-02
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (NINDS)
ID : 1RF1NS118448-01
Investigateurs
Larry Abbot
(L)
Luigi Acerbi
(L)
Valeria Aguillon-Rodriguez
(V)
Mandana Ahmadi
(M)
Jaweria Amjad
(J)
Dora Angelaki
(D)
Jaime Arlandis
(J)
Zoe C Ashwood
(ZC)
Kush Banga
(K)
Hailey Barrell
(H)
Hannah M Bayer
(HM)
Brandon Benson
(B)
Julius Benson
(J)
Jai Bhagat
(J)
Dan Birman
(D)
Niccolò Bonacchi
(N)
Kcenia Bougrova
(K)
Julien Boussard
(J)
Sebastian A Bruijns
(SA)
E Kelly Buchanan
(EK)
Robert Campbell
(R)
Matteo Carandini
(M)
Joana A Catarino
(JA)
Fanny Cazettes
(F)
Gaelle A Chapuis
(GA)
Anne K Churchland
(AK)
Yang Dan
(Y)
Felicia Davatolhagh
(F)
Peter Dayan
(P)
Sophie Denève
(S)
Eric E J DeWitt
(EEJ)
Ling Liang Dong
(LL)
Tatiana Engel
(T)
Michele Fabbri
(M)
Mayo Faulkner
(M)
Robert Fetcho
(R)
Ila Fiete
(I)
Charles Findling
(C)
Laura Freitas-Silva
(L)
Surya Ganguli
(S)
Berk Gercek
(B)
Naureen Ghani
(N)
Ivan Gordeliy
(I)
Laura M Haetzel
(LM)
Kenneth D Harris
(KD)
Michael Hausser
(M)
Naoki Hiratani
(N)
Sonja Hofer
(S)
Fei Hu
(F)
Felix Huber
(F)
Cole Hurwitz
(C)
Anup Khanal
(A)
Christopher S Krasniak
(CS)
Sanjukta Krishnagopal
(S)
Michael Krumin
(M)
Debottam Kundu
(D)
Agnès Landemard
(A)
Christopher Langdon
(C)
Christopher Langfield
(C)
Inês Laranjeira
(I)
Peter Latham
(P)
Petrina Lau
(P)
Hyun Dong Lee
(HD)
Ari Liu
(A)
Zachary F Mainen
(ZF)
Amalia Makri-Cottington
(A)
Hernando Martinez-Vergara
(H)
Brenna McMannon
(B)
Isaiah McRoberts
(I)
Guido T Meijer
(GT)
Maxwell Melin
(M)
Leenoy Meshulam
(L)
Kim Miller
(K)
Nathaniel J Miska
(NJ)
Catalin Mitelut
(C)
Zeinab Mohammadi
(Z)
Thomas Mrsic-Flogel
(T)
Masayoshi Murakami
(M)
Jean-Paul Noel
(JP)
Kai Nylund
(K)
Farideh Oloomi
(F)
Alejandro Pan-Vazquez
(A)
Liam Paninski
(L)
Alberto Pezzotta
(A)
Samuel Picard
(S)
Jonathan W Pillow
(JW)
Alexandre Pouget
(A)
Florian Rau
(F)
Cyrille Rossant
(C)
Noam Roth
(N)
Nicholas A Roy
(NA)
Kamron Saniee
(K)
Rylan Schaeffer
(R)
Michael M Schartner
(MM)
Yanliang Shi
(Y)
Carolina Soares
(C)
Karolina Z Socha
(KZ)
Cristian Soitu
(C)
Nicholas A Steinmetz
(NA)
Karel Svoboda
(K)
Marsa Taheri
(M)
Charline Tessereau
(C)
Anne E Urai
(AE)
Erdem Varol
(E)
Miles J Wells
(MJ)
Steven J West
(SJ)
Matthew R Whiteway
(MR)
Charles Windolf
(C)
Olivier Winter
(O)
Ilana Witten
(I)
Lauren E Wool
(LE)
Zekai Xu
(Z)
Han Yu
(H)
Anthony M Zador
(AM)
Yizi Zhang
(Y)
Informations de copyright
© 2024. The Author(s), under exclusive licence to Springer Nature America, Inc.
Références
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017).
doi: 10.1016/j.neuron.2016.12.041
pubmed: 28182904
Branson, K., Robie, A. A., Bender, J., Perona, P. & Dickinson, M. H. High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451–457 (2009).
doi: 10.1038/nmeth.1328
pubmed: 19412169
pmcid: 2734963
Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. Royal Soc. Interface 11, 20140672 (2014).
doi: 10.1098/rsif.2014.0672
Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).
doi: 10.1016/j.neuron.2015.11.031
pubmed: 26687221
pmcid: 4708087
Wiltschko, A. B. et al. Revealing the structure of pharmacobehavioral space through motion sequencing. Nat. Neurosci. 23, 1433–1443 (2020).
doi: 10.1038/s41593-020-00706-3
pubmed: 32958923
pmcid: 7606807
Luxem, K. et al. Identifying behavioral structure from deep variational embeddings of animal motion. Commun. Biol. 5, 1267 (2022).
doi: 10.1038/s42003-022-04080-7
pubmed: 36400882
pmcid: 9674640
Mathis, A. et al. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).
doi: 10.1038/s41593-018-0209-y
pubmed: 30127430
Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2019).
doi: 10.1038/s41592-018-0234-5
pubmed: 30573820
Graving, J. M. et al. Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife 8, e47994 (2019).
doi: 10.7554/eLife.47994
pubmed: 31570119
pmcid: 6897514
Dunn, T. W. et al. Geometric deep learning enables 3D kinematic profiling across species and environments. Nat. Methods 18, 564–573 (2021).
doi: 10.1038/s41592-021-01106-6
pubmed: 33875887
pmcid: 8530226
Chen, Z. et al. Alphatracker: a multi-animal tracking and behavioral analysis tool. Front. Behav. Neurosci. 17, 1111908 (2023).
doi: 10.3389/fnbeh.2023.1111908
pubmed: 37324523
pmcid: 10266280
Jones, J. M. et al. A machine-vision approach for automated pain measurement at millisecond timescales. Elife 9, e57258 (2020).
doi: 10.7554/eLife.57258
pubmed: 32758355
pmcid: 7434442
Padilla-Coreano, N. et al. Cortical ensembles orchestrate social competition through hypothalamic outputs. Nature 603, 667–671 (2022).
doi: 10.1038/s41586-022-04507-5
pubmed: 35296862
pmcid: 9576144
Warren, R. A. et al. A rapid whisker-based decision underlying skilled locomotion in mice. Elife 10, e63596 (2021).
doi: 10.7554/eLife.63596
pubmed: 33428566
pmcid: 7800376
Hsu, A. I. & Yttri, E. A. B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nat. Commun. 12, 5188 (2021).
doi: 10.1038/s41467-021-25420-x
pubmed: 34465784
pmcid: 8408193
Pereira, T. D. et al. Sleap: a deep learning system for multi-animal pose tracking. Nat. Methods 19, 486–495 (2022).
doi: 10.1038/s41592-022-01426-1
pubmed: 35379947
pmcid: 9007740
Weinreb, C. et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Preprint at bioRxiv https://doi.org/10.1101/2023.03.16.532307 (2023).
Karashchuk, P. et al. Anipose: a toolkit for robust markerless 3D pose estimation. Cell Rep. 36, 109730 (2021).
doi: 10.1016/j.celrep.2021.109730
pubmed: 34592148
pmcid: 8498918
Monsees, A. et al. Estimation of skeletal kinematics in freely moving rodents. Nat. Methods 19, 1500–1509 (2022).
doi: 10.1038/s41592-022-01634-9
pubmed: 36253644
pmcid: 9636019
Rodgers, C. C. A detailed behavioral, videographic, and neural dataset on object recognition in mice. Sci. Data 9, 620 (2022).
doi: 10.1038/s41597-022-01728-1
pubmed: 36229608
pmcid: 9561117
Chapelle, O., Schölkopf, B. & Zien, A. (eds) Semi-Supervised Learning (The MIT Press, 2006).
Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. in Advances in Neural Information Processing Systems vol. 30 (eds Guyon, I. et al.) (Curran Associates, 2017).
Abe, T. et al. Neuroscience cloud analysis as a service: An open-source platform for scalable, reproducible data analysis. Neuron 110, 2771–2789 (2022).
doi: 10.1016/j.neuron.2022.06.018
pubmed: 35870448
pmcid: 9464703
Falcon, W. et al. Pytorchlightning/pytorch-lightning: 0.7.6 release. Zenodo https://doi.org/10.5281/zenodo.3828935 (2020).
Recht, B., Roelofs, R., Schmidt, L. & Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 5389–5400 (PMLR, 2019).
Tran, D. et al. Plex: Towards reliability using pretrained large model extensions. Preprint at https://arxiv.org/abs/2207.07411 (2022).
Burgos-Artizzu, X. P., Dollár, P., Lin, D., Anderson, D. J. & Perona, P. Social behavior recognition in continuous video. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 1322–1329 (IEEE, 2012).
Segalin, C. et al. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. Elife 10, e63720 (2021).
doi: 10.7554/eLife.63720
pubmed: 34846301
pmcid: 8631946
IBL. Data release - Brainwide map - Q4 2022 (2023). Figshare https://doi.org/10.6084/m9.figshare.21400815.v6 (2022).
Desai, N. et al. Openapepose, a database of annotated ape photographs for pose estimation. Elife 12, RP86873 (2023).
doi: 10.7554/eLife.86873
pubmed: 38078902
pmcid: 10712952
Syeda, A. et al. Facemap: a framework for modeling neural activity based on orofacial tracking. Nat. Neurosci. 27, 187–195 (2024).
Spelke, E. S. Principles of object perception. Cogn. Sci. 14, 29–56 (1990).
doi: 10.1207/s15516709cog1401_3
Wu, A. et al. Deep graph pose: a semi-supervised deep graphical model for improved animal pose tracking. in Advances in Neural Information Processing Systems (eds Larochelle, H. et al.) 6040–6052 (2020).
Nath, T. et al. Using deeplabcut for 3D markerless pose estimation across species and behaviors. Nat. Protoc. 14, 2152–2176 (2019).
doi: 10.1038/s41596-019-0176-0
pubmed: 31227823
Zhang, Y. & Park, H. S. Multiview supervision by registration. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 420–428 (2020).
He, Y., Yan, R., Fragkiadaki, K. & Yu, S.-I. Epipolar transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7779–7788 (2020).
Hartley, R. & Zisserman, A. Multiple View Geometry in Computer Vision (Cambridge University Press, 2003).
Bialek, W. On the dimensionality of behavior. Proc. Natl Acad. Sci. uSA 119, e2021860119 (2022).
doi: 10.1073/pnas.2021860119
pubmed: 35486689
pmcid: 9170048
Stephens, G. J., Johnson-Kerner, B., Bialek, W. & Ryu, W. S. From modes to movement in the behavior of caenorhabditis elegans. PloS ONE 5, e13914 (2010).
doi: 10.1371/journal.pone.0013914
pubmed: 21103370
pmcid: 2982830
Yan, Y., Goodman, J. M., Moore, D. D., Solla, S. A. & Bensmaia, S. J. Unexpected complexity of everyday manual behaviors. Nat. Commun. 11, 3564 (2020).
doi: 10.1038/s41467-020-17404-0
pubmed: 32678102
pmcid: 7367296
IBL. Video hardware and software for the international brain laboratory. Figshare https://doi.org/10.6084/m9.figshare.19694452.v1 (2022).
Li, T., Severson, K. S., Wang, F. & Dunn, T. W. Improved 3Dd markerless mouse pose estimation using temporal semi-supervision. Int. J. Comput. Vis. 131, 1389–1405 (2023).
Beluch, W. H., Genewein, T., Nürnberger, A. & Köhler, J. M. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9368–9377 (2018).
Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R. & Cunningham, J. P. Deep ensembles work, but are they necessary? in Advances in Neural Information Processing Systems 35, 33646–33660 (2022).
Bishop, C. M. & Nasrabadi, N. M. Pattern Recognition and Machine Learning, vol. 4 (Springer, 2006).
Yu, H. et al. AP-10K: a benchmark for animal pose estimation in the wild. Preprint at https://arxiv.org/abs/2108.12617 (2021).
Ye, S. et al. SuperAnimal models pretrained for plug-and-play analysis of animal behavior. Preprint at https://arxiv.org/abs/2203.07436 (2022).
Zheng, C. et al. Deep learning-based human pose estimation: a survey. ACM Computing Surveys 56, 1–37 (2023).
doi: 10.1145/3603618
Lin, T. -Y. et al. Microsoft coco: common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings. Vol. 8693, 740–755 (Springer, 2014).
Ionescu, C., Papava, D., Olaru, V. & Sminchisescu, C. Human3. 6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1325–1339 (2013).
doi: 10.1109/TPAMI.2013.248
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G. & Black, M. J. SMPL: a skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries. Vol. 2, 851–866 (2023).
Marshall, J. D., Li, T., Wu, J. H. & Dunn, T. W. Leaving flatland: advances in 3D behavioral measurement. Curr. Opin. Neurobiol. 73, 102522 (2022).
doi: 10.1016/j.conb.2022.02.002
pubmed: 35453000
Günel, S. et al. DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila. Elife 8, e48571 (2019).
doi: 10.7554/eLife.48571
pubmed: 31584428
pmcid: 6828327
Sun, J. J. et al. BKinD-3D: self-supervised 3D keypoint discovery from multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9001–9010 (2023).
Bala, P. C. et al. Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nat. Commun. 11, 4560 (2020).
doi: 10.1038/s41467-020-18441-5
pubmed: 32917899
pmcid: 7486906
Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network. Preprint at https://arxiv.org/abs/1503.02531 (2015).
Lauer, J. et al. Multi-animal pose estimation, identification and tracking with deeplabcut. Nat. Meth. 19, 496–504 (2022).
doi: 10.1038/s41592-022-01443-0
Chettih, S. N., Mackevicius, E. L., Hale, S. & Aronov, D. Barcoding of episodic memories in the hippocampus of a food-caching bird. Cell 187, 1922–1935 (2024).
doi: 10.1016/j.cell.2024.02.032
pubmed: 38554707
IBLet al. Standardized and reproducible measurement of decision-making in mice. Elife 10, e63711 (2021).
doi: 10.7554/eLife.63711
IBL et al. Reproducibility of in vivo electrophysiological measurements in mice. Preprint at bioRxiv https://doi.org/10.1101/2022.05.09.491042 (2022).
Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. in Advances in Neural Information Processing Systems 32, 8024–8035 (2019).
Jafarian, Y., Yao, Y. & Park, H. S. MONET: multiview semi-supervised keypoint via epipolar divergence. Preprint at https://arxiv.org/abs/1806.00104 (2018).
Tresch, M. C. & Jarc, A. The case for and against muscle synergies. Curr. Opin. Neurobiol. 19, 601–607 (2009).
doi: 10.1016/j.conb.2009.09.002
pubmed: 19828310
pmcid: 2818278
Stephens, G. J., Johnson-Kerner, B., Bialek, W. & Ryu, W. S. Dimensionality and dynamics in the behavior of C. elegans. PLoS Comput. Biol. 4, e1000028 (2008).
doi: 10.1371/journal.pcbi.1000028
pubmed: 18389066
pmcid: 2276863
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
Virtanen, P. et al. Scipy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020).
doi: 10.1038/s41592-019-0686-2
pubmed: 32015543
pmcid: 7056644
IBL et al. A brain-wide map of neural activity during complex behaviour. Preprint at bioRxiv https://doi.org/10.1101/2023.07.04.547681 (2023).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Zolnouri, M., Li, X. & Nia, V. P. Importance of data loading pipeline in training deep neural networks. Preprint at https://arxiv.org/abs/2005.02130 (2020).
Yadan, O. Hydra - a framework for elegantly configuring complex applications. Github https://github.com/facebookresearch/hydra (2019).
Whiteway, M, Biderman, D., Warren, R., Zhang, Q. & Sawtell, N. B. Lightning Pose dataset: mirror-mouse. Figshare https://doi.org/10.6084/m9.figshare.24993315.v1 (2024).
Whiteway, M. et al. Lightning Pose dataset: mirror-fish. Figshare https://doi.org/10.6084/m9.figshare.24993363.v1 (2024).
Whiteway, M. & Biderman, D. Lightning Pose dataset: CRIM13. Figshare https://doi.org/10.6084/m9.figshare.24993384.v1 (2024).
Whiteway, M. & Biderman, D. Lightning Pose results: Nature Methods 2024. Figshare https://doi.org/10.6084/m9.figshare.25412248.v2 (2024).