Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
08 Feb 2024
08 Feb 2024
Historique:
received:
27
07
2023
accepted:
25
01
2024
medline:
9
2
2024
pubmed:
9
2
2024
entrez:
8
2
2024
Statut:
epublish
Résumé
Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32[Formula: see text]32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.
Identifiants
pubmed: 38331957
doi: 10.1038/s41598-024-52966-9
pii: 10.1038/s41598-024-52966-9
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
3286Subventions
Organisme : U.S. Department of Energy (DOE)
ID : DE-AC52-07NA27344
Informations de copyright
© 2024. The Author(s).
Références
Datta, R., Heaster, T. M., Sharick, J. T., Gillette, A. A. & Skala, M. C. Fluorescence lifetime imaging microscopy: Fundamentals and advances in instrumentation, analysis, and applications. J. Biomed. Opt. 25, 1–43. https://doi.org/10.1117/1.JBO.25.7.071203 (2020).
doi: 10.1117/1.JBO.25.7.071203
pubmed: 32406215
van Munster, E. B. & Gadella, T. W. J. Fluorescence lifetime imaging microscopy (FLIM). Microsc. Tech. https://doi.org/10.1007/b102213 (2005).
doi: 10.1007/b102213
Suhling, K. et al. Fluorescence lifetime imaging (FLIM): Basic concepts and some recent developments. Med. Photonics 27, 3–40. https://doi.org/10.1016/j.medpho.2014.12.001 (2015).
doi: 10.1016/j.medpho.2014.12.001
Wallrabe, H. & Periasamy, A. Imaging protein molecules using FRET and FLIM microscopy. Curr. Opin. Biotechnol. 16, 19–27. https://doi.org/10.1016/j.copbio.2004.12.002 (2005).
doi: 10.1016/j.copbio.2004.12.002
pubmed: 15722011
Unger, J. et al. Real-time diagnosis and visualization of tumor margins in excised breast specimens using fluorescence lifetime imaging and machine learning. Biomed. Opt. Express 11, 1216–1230. https://doi.org/10.1364/BOE.381358 (2020).
doi: 10.1364/BOE.381358
pubmed: 32206404
pmcid: 7075618
Erkkilä, M. T. et al. Macroscopic fluorescence-lifetime imaging of NADH and protoporphyrin IX improves the detection and grading of 5-aminolevulinic acid-stained brain tumors. Sci. Rep. 10, 20492. https://doi.org/10.1038/s41598-020-77268-8 (2020).
doi: 10.1038/s41598-020-77268-8
pubmed: 33235233
pmcid: 7686506
Weyers, B. W. et al. Fluorescence lifetime imaging for intraoperative cancer delineation in transoral robotic surgery. Transl. Biophotonics https://doi.org/10.1002/tbio.201900017 (2019).
doi: 10.1002/tbio.201900017
pubmed: 32656529
pmcid: 7351319
Phipps, J., et al. Head and neck cancer evaluation via transoral robotic surgery with augmented fluorescence lifetime imaging. In Biophotonics Congress: Biomedical Optics Congress 2018 (Microscopy/Translational/Brain/OTS), CTu2B.3, 10.1364/TRANSLATIONAL.2018.CTu2B.3 (OSA, Washington, D.C.).
Becker, W. TCSPC Handbook 9th edn. (2021).
Becker, W. Fluorescence lifetime imaging-techniques and applications. J. Microsc. 247, 119–136. https://doi.org/10.1111/j.1365-2818.2012.03618.x (2012).
doi: 10.1111/j.1365-2818.2012.03618.x
pubmed: 22621335
Hirvonen, L. M. & Suhling, K. Wide-field TCSPC: Methods and applications. Meas. Sci. Technol. 28, 012003. https://doi.org/10.1088/1361-6501/28/1/012003 (2017).
doi: 10.1088/1361-6501/28/1/012003
Kapusta, P., Wahl, M. & Erdmann, R. Advanced photon counting: Applications, methods, instrumentation/volume editors, Peter Kapusta, Michael Wahl, Rainer Erdmann; with contributions by A. Ahlrichs [and fifty others], vol. 15 of Springer Series on Fluorescence, 1617-1306 (Springer, Cham, 2015).
Alfonso-Garcia, A. et al. Mesoscopic fluorescence lifetime imaging: Fundamental principles, clinical applications and future directions. J. Biophotonics 14, e202000472. https://doi.org/10.1002/jbio.202000472 (2021).
doi: 10.1002/jbio.202000472
pubmed: 33710785
pmcid: 8579869
Morimoto, K. Megapixel SPAD cameras for time-resolved applications, https://doi.org/10.5075/EPFL-THESIS-8773 .
Caccia, M., Nardo, L., Santoro, R. & Schaffhauser, D. Silicon photomultipliers and SPAD imagers in biophotonics: Advances and perspectives. Nucl. Instrum. Methods Phys. Res., Sect. A 926, 101–117. https://doi.org/10.1016/j.nima.2018.10.204 (2019).
doi: 10.1016/j.nima.2018.10.204
Bruschini, C., Homulle, H., Antolovic, I. M., Burri, S. & Charbon, E. Single-photon avalanche diode imagers in biophotonics: Review and outlook. Light Sci. Appl. 8, 87. https://doi.org/10.1038/s41377-019-0191-5 (2019).
doi: 10.1038/s41377-019-0191-5
pubmed: 31645931
pmcid: 6804596
Cusini, I. et al. Historical perspectives, state of art and research trends of SPAD arrays and their applications (part ii: SPAD arrays). Front. Phys. 10, 606. https://doi.org/10.3389/fphy.2022.906671 (2022).
doi: 10.3389/fphy.2022.906671
Grinvald, A. & Steinberg, I. Z. On the analysis of fluorescence decay kinetics by the method of least-squares. Anal. Biochem. 59, 583–598. https://doi.org/10.1016/0003-2697(74)90312-1 (1974).
doi: 10.1016/0003-2697(74)90312-1
pubmed: 4838786
Bajzer, E., Therneau, T. M., Sharp, J. C. & Prendergast, F. G. Maximum likelihood method for the analysis of time-resolved fluorescence decay curves. Eur. Biophys. J. 20, 247–262. https://doi.org/10.1007/BF00450560 (1991).
doi: 10.1007/BF00450560
Chessel, A., Waharte, F., Salamero, J. & Kervrann, C. A maximum likelihood method for lifetime estimation in photon counting-based fluorescence lifetime imaging microscopy. In 21st European Signal Processing Conference (EUSIPCO 2013), 1–5 (2013).
Isenberg, I. & Dyson, R. D. The analysis of fluorescence decay by a method of moments. Biophys. J . 9, 1337–1350. https://doi.org/10.1016/S0006-3495(69)86456-8 (1969).
doi: 10.1016/S0006-3495(69)86456-8
pubmed: 5353139
pmcid: 1367635
Li, D.-U. et al. Real-time fluorescence lifetime imaging system with a 32[Formula: see text]32 0.13 [Formula: see text]m CMOS low dark-count single-photon avalanche diode array. Opt. Express https://doi.org/10.1364/OE.18.010257 (2010).
doi: 10.1364/OE.18.010257
pubmed: 21197050
pmcid: 3090649
Li, D.D.-U. et al. Video-rate fluorescence lifetime imaging camera with CMOS single-photon avalanche diode arrays and high-speed imaging algorithm. J. Biomed. Opt. 16, 096012. https://doi.org/10.1117/1.3625288 (2011).
doi: 10.1117/1.3625288
pubmed: 21950926
Liu, X. et al. Fast fluorescence lifetime imaging techniques: A review on challenge and development. J. Innov. Opt. Health Sci. 12, 1930003. https://doi.org/10.1142/S1793545819300039 (2019).
doi: 10.1142/S1793545819300039
Mannam, V., Zhang, Y., Yuan, X., Ravasio, C. & Howard, S. S. Machine learning for faster and smarter fluorescence lifetime imaging microscopy. J. Phys. Photonics 2, 042005. https://doi.org/10.1088/2515-7647/abac1a (2020).
doi: 10.1088/2515-7647/abac1a
Wu, G., Nowotny, T., Zhang, Y., Yu, H.-Q. & Li, D.D.-U. Artificial neural network approaches for fluorescence lifetime imaging techniques. Opt. Lett. 41, 2561–2564. https://doi.org/10.1364/OL.41.002561 (2016).
doi: 10.1364/OL.41.002561
pubmed: 27244414
Smith, J. T. et al. Fast fit-free analysis of fluorescence lifetime imaging via deep learning. Proc. Natl. Acad. Sci. 116, 24019–24030. https://doi.org/10.1073/pnas.1912707116 (2019).
doi: 10.1073/pnas.1912707116
pubmed: 31719196
pmcid: 6883809
Zickus, V. et al. Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation. Sci. Rep. 10, 20986. https://doi.org/10.1038/s41598-020-77737-0 (2020).
doi: 10.1038/s41598-020-77737-0
pubmed: 33268900
pmcid: 7710711
Xiao, D., Chen, Y. & Li, D.D.-U. One-dimensional deep learning architecture for fast fluorescence lifetime imaging. IEEE J. Sel. Top. Quantum Electron. 27, 1–10. https://doi.org/10.1109/JSTQE.2021.3049349 (2021).
doi: 10.1109/JSTQE.2021.3049349
pubmed: 33154613
Zang, Z., et al. Fast fluorescence lifetime imaging analysis via extreme learning machine. arXiv preprint arXiv:2203.13754 10.48550/arXiv.2203.13754 (2022).
Chen, Y.-I. et al. Generative adversarial network enables rapid and robust fluorescence lifetime image analysis in live cells. Commun. Biol. 5, 18. https://doi.org/10.1038/s42003-021-02938-w (2022).
doi: 10.1038/s42003-021-02938-w
pubmed: 35017629
pmcid: 8752789
Marsden, M. et al. Intraoperative margin assessment in oral and oropharyngeal cancer using label-free fluorescence lifetime imaging and machine learning. I.E.E.E. Trans. Biomed. Eng. 68, 857–868. https://doi.org/10.1109/TBME.2020.3010480 (2021).
doi: 10.1109/TBME.2020.3010480
Sagar, M. A. K. et al. Machine learning methods for fluorescence lifetime imaging (FLIM) based label-free detection of microglia. Front. Neurosci. 14, 931. https://doi.org/10.3389/fnins.2020.00931 (2020).
doi: 10.3389/fnins.2020.00931
pubmed: 33013309
pmcid: 7497798
Xiao, D. et al. Dynamic fluorescence lifetime sensing with CMOS single-photon avalanche diode arrays and deep learning processors. Biomed. Opt. Express 12, 3450–3462. https://doi.org/10.1364/BOE.425663 (2021).
doi: 10.1364/BOE.425663
pubmed: 34221671
pmcid: 8221960
Zang, Z. et al. Compact and robust deep learning architecture for fluorescence lifetime imaging and FPGA implementation. Methods Appl. Fluoresc. 11, 025002. https://doi.org/10.1088/2050-6120/acc0d9 (2023).
doi: 10.1088/2050-6120/acc0d9
Xiao, D., et al. Smart Wide-field Fluorescence Lifetime Imaging System with CMOS Single-photon Avalanche Diode Arrays. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference 2022, 1887–1890, 10.1109/EMBC48229.2022.9870996 (2022).
Elman, J. L. Finding structure in time. Cogn. Sci. 14, 179–211. https://doi.org/10.1207/s15516709cog1402_1 (1990).
doi: 10.1207/s15516709cog1402_1
Zhang, C., Lindner, S., Antolovic, I. M., Wolf, M. & Charbon, E. A CMOS SPAD imager with collision detection and 128 dynamically reallocating TDCs for single-photon counting and 3D time-of-flight imaging. Sensors 18, 4016. https://doi.org/10.3390/s18114016 (2018).
doi: 10.3390/s18114016
pubmed: 30453648
pmcid: 6263909
Kalyanov, A., Ackermann, M., Russomanno, E., Wolf, M. & Jiang, J. Time-multiplexing approach for fast time-domain near-infrared optical tomography combined with neural-network-enhanced image reconstruction. In Diffuse Optical Spectroscopy and Imaging IX12628, 56–58. https://doi.org/10.1117/12.2670186 (SPIE, 2023).
Cho, K., et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 10.48550/arXiv.1406.1078 (2014).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 (1997).
doi: 10.1162/neco.1997.9.8.1735
pubmed: 9377276
Bouchet, D., Krachmalnicoff, V. & Izeddin, I. Cramér-Rao analysis of lifetime estimations in time-resolved fluorescence microscopy. Opt. Express 27, 21239–21252. https://doi.org/10.1364/OE.27.021239 (2019).
doi: 10.1364/OE.27.021239
pubmed: 31510207
Lindner, S., et al. A novel 32 x 32, 224 Mevents/s time resolved SPAD image sensor for near-infrared optical tomography. Microscopy Histopathology and Analytics JTh5A.6, 10.1364/TRANSLATIONAL.2018.JTh5A.6 (2018).
Charbon, E., Bruschini, C. & Lee, M.-J. 3D-stacked CMOS SPAD image sensors: Technology and applications. in 2018 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS), 1–4, https://doi.org/10.1109/ICECS.2018.8617983 (IEEE, 12/9/2018 - 12/12/2018).
Xiao, D. et al. Spatial resolution improved fluorescence lifetime imaging via deep learning. Opt. Express 30, 11479–11494. https://doi.org/10.1364/OE.451215 (2022).
doi: 10.1364/OE.451215
pubmed: 35473091
Héliot, L. & Leray, A. Simple phasor-based deep neural network for fluorescence lifetime imaging microscopy. Sci. Rep. 11, 23858. https://doi.org/10.1038/s41598-021-03060-x (2021).
doi: 10.1038/s41598-021-03060-x
pubmed: 34903737
pmcid: 8668934
Yao, R., Ochoa, M., Yan, P. & Intes, X. Net-FLICS: Fast quantitative wide-field fluorescence lifetime imaging with compressed sensing—A deep learning approach. Light Sci. Appl. 8, 26. https://doi.org/10.1038/s41377-019-0138-x (2019).
doi: 10.1038/s41377-019-0138-x
pubmed: 30854198
pmcid: 6400960
PolyAn GmbH. Fluorescence lifetime beads. https://www.poly-an.de/micro-nanoparticles/other-microparticles/fluorescence-lifetime-applications . Accessed: 2023-10-16.
Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. https://doi.org/10.48550/arXiv.1912.01703 (2019).
doi: 10.48550/arXiv.1912.01703
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. in Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp 249–256 (JMLR Workshop and Conference Proceedings, 2010).
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 10.48550/arXiv.1412.6980 (2014).
Köllner, M. & Wolfrum, J. How many photons are necessary for fluorescence-lifetime measurements?. Chem. Phys. Lett. 200, 199–204. https://doi.org/10.1016/0009-2614(92)87068-Z (1992).
doi: 10.1016/0009-2614(92)87068-Z
Kim, J. & Seok, J. Statistical properties of amplitude and decay parameter estimators for fluorescence lifetime imaging. Opt. Express 21, 6061–6075. https://doi.org/10.1364/OE.21.006061 (2013).
doi: 10.1364/OE.21.006061
pubmed: 23482174