Multi-resolution convolutional neural networks for inverse problems.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
31 03 2020
31 03 2020
Historique:
received:
23
12
2019
accepted:
13
03
2020
entrez:
3
4
2020
pubmed:
3
4
2020
medline:
3
4
2020
Statut:
epublish
Résumé
Inverse problems in image processing, phase imaging, and computer vision often share the same structure of mapping input image(s) to output image(s) but are usually solved by different application-specific algorithms. Deep convolutional neural networks have shown great potential for highly variable tasks across many image-based domains, but sometimes can be challenging to train due to their internal non-linearity. We propose a novel, fast-converging neural network architecture capable of solving generic image(s)-to-image(s) inverse problems relevant to a diverse set of domains. We show this approach is useful in recovering wavefronts from direct intensity measurements, imaging objects from diffusely reflected images, and denoising scanning transmission electron microscopy images, just by using different training datasets. These successful applications demonstrate the proposed network to be an ideal candidate solving general inverse problems falling into the category of image(s)-to-image(s) translation.
Identifiants
pubmed: 32235861
doi: 10.1038/s41598-020-62484-z
pii: 10.1038/s41598-020-62484-z
pmc: PMC7109091
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
5730Références
Russakovsky, O. et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 211–252 (2015).
doi: 10.1007/s11263-015-0816-y
Krasin, I.et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. 2, 7 (2016).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
doi: 10.1038/nature14539
Dong, C., Loy, C. C., He, K. & Tang, X. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 295–307 (2016).
doi: 10.1109/TPAMI.2015.2439281
Zhang, K., Zuo, W., Chen, Y., Meng, D. & Zhang, L. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Transactions on Image Processing 26, 3142–3155 (2017).
doi: 10.1109/TIP.2017.2662206
Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. CVPR (2017).
Hezaveh, Y. D., Levasseur, L. P. & Marshall, P. J. Fast automated analysis of strong gravitational lenses with convolutional neural networks. Nature 548, 555–557 (2017).
doi: 10.1038/nature23463
Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).
doi: 10.1364/OPTICA.4.001117
Hornik, K., Stinchcombe, M. & White, H. Multilayer feedforward networks are universal approximators. Neural Networks 2, 359–366 (1989).
doi: 10.1016/0893-6080(89)90020-8
Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2, 303–314 (1989).
doi: 10.1007/BF02551274
Lu, Z., Pu, H., Wang, F., Hu, Z. & Wang, L. The Expressive Power of Neural Networks: A View from the Width. In Guyon, I. et al. (eds.) Advances in Neural Information Processing Systems 30, 6231–6239 (Curran Associates, Inc., 2017).
Yang, Z. et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237 (2019).
Brock, A., Donahue, J. & Simonyan, K. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv:1809.11096 (2018).
Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging 6, 921–943, https://www.osapublishing.org/optica/abstract.cfm?uri=optica-6-8-921 .
Jin, K. H., McCann, M. T., Froustey, E. & Unser, M. Deep Convolutional Neural Network for Inverse Problems in Imaging. In IEEE Transactions on Image Processing, 26, 4509–4522, https://doi.org/10.1109/TIP.2017.2713099 (2017).
Luo, R., Tian, F., Qin, T., Chen, E. & Liu, T.-Y. Neural architecture optimization. In Bengio, S. et al. (eds.) Advances in Neural Information Processing Systems 31, 7816–7827 (Curran Associates, Inc., 2018).
Jin, H., Song, Q. & Hu, X. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1946–1956 (ACM, 2019).
Smith, L. N. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. arXiv:1803.09820 (2018).
Xu, Z.-Q. J., Zhang, Y. & Xiao, Y. Training behavior of deep neural network in frequency domain. arXiv preprint arXiv:1807.01251 (2018).
Rahaman, N. et al. On the Spectral Bias of Deep Neural Networks. arXiv preprint arXiv:1806.08734 (2018).
Stüben, K. & Trottenberg, U. Multigrid methods: Fundamental algorithms, model problem analysis and applications. In Multigrid Methods, 1–176 (Springer, 1982).
Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. science 313, 504–507 (2006).
doi: 10.1126/science.1127647
Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (Springer International Publishing, 2015).
Karras, T., Aila, T., Laine, S. & Lehtinen, J. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv:1710.10196 (2017).
Nah, S., Kim, T. H. & Lee, K. M. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017).
Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 448–456 (JMLR.org, 2015).
Wu, Y. & He, K. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 3–19 (2018).
Gabor, D. A New Microscopic Principle. Nature 161, 777–778 (1948).
doi: 10.1038/161777a0
Teague, M. R. Deterministic phase retrieval: A Green’s function solution. JOSA 73, 1434–1441 (1983).
doi: 10.1364/JOSA.73.001434
Van Dyck, D., Op de Beeck, M. & Coene, W. Object wavefunction reconstruction in high resolution electron microscopy. In Proceedings of 1st International Conference on Image Processing, vol. 3, 295–298 vol.3 (1994).
Jingshan, Z., Claus, R. A., Dauwels, J., Tian, L. & Waller, L. Transport of Intensity phase imaging by intensity spectrum fitting of exponentially spaced defocus planes. Optics Express 22, 10661 (2014).
doi: 10.1364/OE.22.010661
Eljarrat, A., Müller, J., Huang, M. R. S. & Koch, C. T. Multi-focus TIE algorithm including partial spatial coherence and overlapping filters. Optics Express 26, 11819 (2018).
doi: 10.1364/OE.26.011819
Nugent, K. A., Gureyev, T. E., Cookson, D., Paganin, D. & Barnea, Z. Quantitative phase imaging using hard x-rays. Phys. Rev. Lett. 77, 2961–2964 (1996).
doi: 10.1103/PhysRevLett.77.2961
Parvizi, A., Van den Broek, W. & Koch, C. T. Recovering low spatial frequencies in wavefront sensing based on intensity measurements. Advanced Structural and Chemical Imaging 2, 1 (2017).
Rivenson, Y., Zhang, Y., Günaydin, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light: Science & Applications 7, 17141 (2018).
doi: 10.1038/lsa.2017.141
Goy, A., Arthur, K., Li, S. & Barbastathis, G. Low Photon Count Phase Retrieval Using Deep Learning. Physical Review Letters 121, 243902, https://doi.org/10.1103/PhysRevLett.121.243902 (2018).
doi: 10.1103/PhysRevLett.121.243902
pubmed: 30608745
pmcid: 30608745
Cherukara, M. J., Nashed, Y. S. G. & Harder, R. J. Real-time coherent diffraction inversion using deep generative networks. Scientific Reports 8, 16520, http://www.nature.com/articles/s41598-018-34525-1 (2018).
Klein, J., Peters, C., Martìn, J., Laurenzis, M. & Hullin, M. B. Tracking objects outside the line of sight using 2d intensity images. Scientific Reports 6, 32491 (2016).
doi: 10.1038/srep32491
O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338–341 (2018).
doi: 10.1038/nature25489
Saunders, C., Murray-Bruce, J. & Goyal, V. K. Computational periscopy with an ordinary digital camera. Nature 565, 472–475 (2019).
doi: 10.1038/s41586-018-0868-6
Satat, G., Tancik, M., Gupta, O., Heshmat, B. & Raskar, R. Object classification through scattering media with deep learning on time resolved measurement. Optics Express 25, 17466–17479, https://www.osapublishing.org/oe/abstract.cfm?uri=oe-25-15-17466 (2017).
Blender Foundation. Big buck bunny, http://bigbuckbunny.org/ (2008).
Erni, R., Rossell, M. D., Kisielowski, C. & Dahmen, U. Atomic-resolution imaging with a sub-50-pm electron probe. Physical review letters 102, 096101 (2009).
doi: 10.1103/PhysRevLett.102.096101
Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-Ångström resolution. Nature 559, 343–349 (2018).
doi: 10.1038/s41586-018-0298-5
Henninen, T. R., Bon, M., Wang, F., Passerone, D. & Erni, R. The Structure of Sub-nm Platinum Clusters at Elevated Temperatures. Angewandte Chemie International Edition, https://doi.org/10.1002/anie.201911068 (2019).
Mevenkamp, N. et al. Poisson noise removal from high-resolution STEM images based on periodic block matching. Advanced Structural and Chemical Imaging 1, 3 (2015-03-25).
Furnival, T., Leary, R. K. & Midgley, P. A. Denoising time-resolved microscopy image sequences with singular value thresholding. Ultramicroscopy 178, 112–124 (2017).
doi: 10.1016/j.ultramic.2016.05.005
Guo, S., Yan, Z., Zhang, K., Zuo, W. & Zhang, L. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1712–1722 (2019).
Lefkimmiatis, S. Universal denoising networks: a novel CNN architecture for image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3204–3213 (2018).
Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9446–9454 (2018).
Wang, P., Zhang, H. & Patel, V. M. SAR Image Despeckling Using a Convolutional Neural Network. IEEE Signal Processing Letters 24, 1763–1767, http://ieeexplore.ieee.org/document/8053792/ (2017).
Hasinoff, S. W. et al. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (TOG) 35, 192 (2016).
doi: 10.1145/2980179.2980254
Xie, J., Xu, L. & Chen, E. Image Denoising and Inpainting with Deep Neural Networks. In Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems 25, 341–349 (Curran Associates, Inc., 2012).
Lehtinen, J. et al. Noise2noise: Learning Image Restoration without Clean Data. arXiv preprint arXiv:1803.04189 (2018).
Batson, J. & Royer, L. Noise2self: Blind Denoising by Self-Supervision. arXiv:1901.11365 [cs, stat], http://arxiv.org/abs/1901.11365 . ArXiv: 1901.11365 (2019).
Krull, A., Buchholz, T.-O. & Jug, F. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2129–2137 (2019).
Eljarrat, A. Python code for inline holography using through-focus image series, https://github.com/AEljarrat/inline_holo (2018).
Krasin, I. et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from, https://github.com/openimages 2, 3 (2017).
Xie, S., Girshick, R., Dollàr, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1492–1500 (2017).
Chollet, F. Xception: Deep Learning with Depthwise Separable Convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1800–1807 (IEEE, Honolulu, HI, 2017).
Goodfellow, I. et al. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems 27, 2672–2680 (Curran Associates, Inc., 2014).
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, http://arxiv.org/abs/1412.6980 (2015).