Stereoscopic depth perception through foliage.
Aerial imaging
Occlusion removal
Stereoscopic depth perception
Synthetic aperture sensing
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
04 Oct 2024
04 Oct 2024
Historique:
received:
09
11
2023
accepted:
27
09
2024
medline:
5
10
2024
pubmed:
5
10
2024
entrez:
4
10
2024
Statut:
epublish
Résumé
Both humans and computational methods struggle to discriminate the depths of objects hidden beneath foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with the human ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, and early wildfire detection, depth assists in differentiating true from false findings, such as people, animals, or vehicles vs. sun-heated patches at the ground level or in the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense woodland to test users' ability to discriminate depth. We found that this is impossible when viewing monoscopic video and relying on motion parallax. The same was true with stereoscopic video because of the occlusions caused by foliage. However, when synthetic aperture sensing was used to reduce occlusions and disparity-scaled stereoscopic video was presented, whereas computational (stereoscopic matching) methods were unsuccessful, human observers successfully discriminated depth. This shows the potential of systems which exploit the synergy between computational methods and human vision to perform tasks that neither can perform alone.
Identifiants
pubmed: 39367044
doi: 10.1038/s41598-024-74666-0
pii: 10.1038/s41598-024-74666-0
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
23056Subventions
Organisme : Austrian Science Fund (FWF)
ID : P32185-NBL
Organisme : German Research Foundation (DFG)
ID : I 6046-N,
Organisme : LIT-Linz Institute of Technology
ID : LIT2019-8-SEE114
Informations de copyright
© 2024. The Author(s).
Références
Kurmi, I., Schedl, D. C. & Bimber, O. Airborne optical sectioning. Journal of. Imaging4, 102 (2018).
doi: 10.3390/jimaging4080102
Bimber, O., Kurmi, I. & Schedl, D. C. Synthetic aperture imaging with drones. IEEE computer graphics and applications39, 8–15 (2019).
doi: 10.1109/MCG.2019.2896024
pubmed: 31021742
Kurmi, I., Schedl, D. C. & Bimber, O. A statistical view on synthetic aperture imaging for occlusion removal. IEEE Sensors Journal19, 9374–9383 (2019).
doi: 10.1109/JSEN.2019.2922731
Kurmi, I., Schedl, D. C. & Bimber, O. Thermal airborne optical sectioning. Remote Sensing11, 1668 (2019).
doi: 10.3390/rs11141668
Schedl, D. C., Kurmi, I. & Bimber, O. Airborne optical sectioning for nesting observation. Scientific reports10, 7254 (2020).
doi: 10.1038/s41598-020-63317-9
pubmed: 32350304
pmcid: 7190638
Kurmi, I., Schedl, D. C. & Bimber, O. Fast automatic visibility optimization for thermal synthetic aperture visualization. IEEE Geoscience and Remote Sensing Letters18, 836–840 (2020).
doi: 10.1109/LGRS.2020.2987471
Kurmi, I., Schedl, D. C. & Bimber, O. Pose error reduction for focus enhancement in thermal synthetic aperture visualization. IEEE Geoscience and Remote Sensing Letters19, 1–5 (2021).
doi: 10.1109/LGRS.2021.3051718
Schedl, D. C., Kurmi, I. & Bimber, O. Search and rescue with airborne optical sectioning. Nature Machine Intelligence2, 783–790 (2020).
doi: 10.1038/s42256-020-00261-3
Schedl, D. C., Kurmi, I. & Bimber, O. An autonomous drone for search and rescue in forests using airborne optical sectioning. Science Robotics6, eabg1188 (2021).
Ortner, R., Kurmi, I. & Bimber, O. Acceleration-aware path planning with waypoints. Drones5, 143 (2021).
Kurmi, I., Schedl, D. C. & Bimber, O. Combined person classification with airborne optical sectioning. Scientific reports12, 3804 (2022).
doi: 10.1038/s41598-022-07733-z
pubmed: 35264622
pmcid: 8907346
Amala Arokia Nathan, R. J., Kurmi, I., Schedl, D. C. & Bimber, O. Through-foliage tracking with airborne optical sectioning. Journal of Remote Sensing2022 (2022).
Seits, F., Kurmi, I., Nathan, R. J. A. A., Ortner, R. & Bimber, O. On the role of field of view for occlusion removal with airborne optical sectioning. arXiv preprint[SPACE] arXiv:2204.13371 (2022).
Amala Arokia Nathan, R. J., Kurmi, I. & Bimber, O. Inverse airborne optical sectioning. Drones6, 231 (2022).
Seits, F., Kurmi, I. & Bimber, O. Evaluation of color anomaly detection in multispectral images for synthetic aperture sensing. Eng3, 541–553 (2022).
doi: 10.3390/eng3040038
Amala Arokia Nathan, R. J., Kurmi, I. & Bimber, O. Drone swarm strategy for the detection and tracking of occluded targets in complex environments. Communications Engineering2, 55 (2023).
Amala Arokia Nathan, R. J. & Bimber, O. Synthetic aperture anomaly imaging for through-foliage target detection. Remote Sensing15, https://doi.org/10.3390/rs15184369 (2023).
Moreira, A. et al. A tutorial on synthetic aperture radar. IEEE Geoscience and remote sensing magazine1, 6–43 (2013).
doi: 10.1109/MGRS.2013.2248301
Li, C. J. & Ling, H. Synthetic aperture radar imaging using a small consumer drone. In 2015 IEEE international symposium on antennas and propagation & USNC/URSI national radio science meeting, 685–686 (IEEE, 2015).
Rosen, P. A. et al. Synthetic aperture radar interferometry. Proceedings of the IEEE88, 333–382 (2000).
doi: 10.1109/5.838084
Levanda, R. & Leshem, A. Synthetic aperture radio telescopes. IEEE Signal Processing Magazine27, 14–29 (2010).
doi: 10.1109/MSP.2009.934719
Dravins, D., Lagadec, T. & Nuñez, P. D. Optical aperture synthesis with electronically connected telescopes. Nature communications6, 6852 (2015).
doi: 10.1038/ncomms7852
pubmed: 25880705
Ralston, T. S., Marks, D. L., Carney, P. S. & Boppart, S. A. Interferometric synthetic aperture microscopy. Nature Physics3, 129–134 (2007).
doi: 10.1038/nphys514
pubmed: 25635181
pmcid: 4308056
Hayes, M. P. & Gough, P. T. Synthetic aperture sonar: A review of current status. IEEE journal of oceanic engineering34, 207–224 (2009).
doi: 10.1109/JOE.2009.2020853
Hansen, R. E. Introduction to synthetic aperture sonar. In Kolev, N. Z. (ed.) Sonar Systems, chap. 1, https://doi.org/10.5772/23122 (IntechOpen, Rijeka, 2011).
Jensen, J. A., Nikolov, S. I., Gammelmark, K. L. & Pedersen, M. H. Synthetic aperture ultrasound imaging. Ultrasonics44, e5–e15 (2006).
doi: 10.1016/j.ultras.2006.07.017
pubmed: 16959281
Zhang, H. K. et al. Synthetic tracked aperture ultrasound imaging: design, simulation, and experimental evaluation. Journal of Medical Imaging3, 027001–027001 (2016).
doi: 10.1117/1.JMI.3.2.027001
pubmed: 27088108
pmcid: 4824841
Barber, Z. W. & Dahl, J. R. Synthetic aperture ladar imaging demonstrations and information at very low return levels. Applied optics53, 5531–5537 (2014).
doi: 10.1364/AO.53.005531
pubmed: 25321130
Turbide, S., Marchese, L., Terroux, M. & Bergeron, A. Synthetic aperture lidar as a future tool for earth observation. In International Conference on Space Optics?ICSO 2014, vol. 10563, 1115–1122 (SPIE, 2017).
Zhang, H., Jin, X. & Dai, Q. Synthetic aperture based on plenoptic camera for seeing through occlusions. In Advances in Multimedia Information Processing–PCM 2018: 19th Pacific-Rim Conference on Multimedia, Hefei, China, September 21-22, 2018, Proceedings, Part I 19, 158–167 (Springer, 2018).
Yang, T. et al. Kinect based real-time synthetic aperture imaging through occlusion. Multimedia Tools and Applications75, 6925–6943 (2016).
doi: 10.1007/s11042-015-2618-1
Pei, Z. et al. Occluded-object 3d reconstruction using camera array synthetic aperture imaging. Sensors19, 607 (2019).
doi: 10.3390/s19030607
pubmed: 30709046
pmcid: 6386989
Burt, P. & Julesz, B. A disparity gradient limit for binocular fusion. Science208, 615–617, https://doi.org/10.1126/science.7367885 (1980). https://www.science.org/doi/pdf/10.1126/science.7367885 .
Deepa, B., Valarmathi, A. & Benita, S. Assessment of stereo acuity levels using random dot stereo acuity chart in college students. Journal of family medicine and primary care8, 3850–3853 (2019).
doi: 10.4103/jfmpc.jfmpc_755_19
pubmed: 31879624
pmcid: 6924232
Filippini, H. R. & Banks, M. S. Limits of stereopsis explained by local cross-correlation. Journal of Vision9, 1–18 (2009).
doi: 10.1167/9.1.8
Tyler, C. W. Depth perception in disparity gratings. Nature251, 140–142 (1974).
doi: 10.1038/251140a0
pubmed: 4420707
Hainich, R. R. & Bimber, O. Displays: fundamentals & applications (CRC press, 2016).
Howard, I. P. Seeing in depth, Vol. 1: Basic mechanisms. (University of Toronto Press, 2002).
Howard, H. J. A test for the judgment of distance. Transactions of the American Ophthalmological Society17, 195 (1919).
pubmed: 16692470
pmcid: 1318185
McKee, S. P. & Verghese, P. Stereo transparency and the disparity gradient limit. Vision Research42, 1963–1977. https://doi.org/10.1016/S0042-6989(02)00073-1 (2002).
doi: 10.1016/S0042-6989(02)00073-1
pubmed: 12160569
Treisman, A. Binocular rivalry and stereoscopic depth perception. Quarterly Journal of Experimental Psychology14, 23–37 (1962).
doi: 10.1080/17470216208416507
Wilcox, L. M. & Lakra, D. C. Depth from binocular half-occlusions in stereoscopic images of natural scenes. Perception36, 830–839 (2007).
doi: 10.1068/p5708
pubmed: 17718362
Forte, J., Peirce, J. W. & Lennie, P. Binocular integration of partially occluded surfaces. Vision Research42, 1225–1235. https://doi.org/10.1016/S0042-6989(02)00053-6 (2002).
doi: 10.1016/S0042-6989(02)00053-6
pubmed: 12044755
Halpern, D. L. & Blake, R. R. How contrast affects stereoacuity. Perception17, 483–495 (1988).
doi: 10.1068/p170483
pubmed: 3244521
Nardini, M., Bedford, R. & Mareschal, D. Fusion of visual cues is not mandatory in children. Proceedings of the National Academy of Sciences107, 17041–17046 (2010).
doi: 10.1073/pnas.1001699107
Zhang, K. et al. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence44, 6360–6376 (2021).
doi: 10.1109/TPAMI.2021.3088914