Blind people can actively manipulate virtual objects with a novel tactile device.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
21 Dec 2023
21 Dec 2023
Historique:
received:
19
05
2023
accepted:
08
12
2023
medline:
22
12
2023
pubmed:
22
12
2023
entrez:
21
12
2023
Statut:
epublish
Résumé
Frequently in rehabilitation, visually impaired persons are passive agents of exercises with fixed environmental constraints. In fact, a printed tactile map, i.e. a particular picture with a specific spatial arrangement, can usually not be edited. Interaction with map content, instead, facilitates the learning of spatial skills because it exploits mental imagery, manipulation and strategic planning simultaneously. However, it has rarely been applied to maps, mainly because of technological limitations. This study aims to understand if visually impaired people can autonomously build objects that are completely virtual. Specifically, we investigated if a group of twelve blind persons, with a wide age range, could exploit mental imagery to interact with virtual content and actively manipulate it by means of a haptic device. The device is mouse-shaped and designed to jointly perceive, with one finger only, local tactile height and inclination cues of arbitrary scalar fields. Spatial information can be mentally constructed by integrating local tactile cues, given by the device, with global proprioceptive cues, given by hand and arm motion. The experiment consisted of a bi-manual task, in which one hand explored some basic virtual objects and the other hand acted on a keyboard to change the position of one object in real-time. The goal was to merge basic objects into more complex objects, like a puzzle. The experiment spanned different resolutions of the tactile information. We measured task accuracy, efficiency, usability and execution time. The average accuracy in solving the puzzle was 90.5%. Importantly, accuracy was linearly predicted by efficiency, measured as the number of moves needed to solve the task. Subjective parameters linked to usability and spatial resolutions did not predict accuracy; gender modulated the execution time, with men being faster than women. Overall, we show that building purely virtual tactile objects is possible in absence of vision and that the process is measurable and achievable in partial autonomy. Introducing virtual tactile graphics in rehabilitation protocols could facilitate the stimulation of mental imagery, a basic element for the ability to orient in space. The behavioural variable introduced in the current study can be calculated after each trial and therefore could be used to automatically measure and tailor protocols to specific user needs. In perspective, our experimental setup can inspire remote rehabilitation scenarios for visually impaired people.
Identifiants
pubmed: 38129483
doi: 10.1038/s41598-023-49507-1
pii: 10.1038/s41598-023-49507-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
22845Informations de copyright
© 2023. The Author(s).
Références
Downs, R. M. & Stea, D. Image and Environment: Cognitive Mapping and Spatial Behavior (Transaction Publishers, 1973).
Ungar, S. Cognitive mapping without visual experience. In Cognitive Mapping: Past Present and Future. (Eds Kitchin, R. & Freundschuh, S.) 221–248 (Routledge, London, 2018).
Noordzij, M. L., Zuidhoek, S. & Postma, A. The influence of visual experience on the ability to form spatial mental models based on route and survey descriptions. Cognition 100, 321–342 (2006).
pubmed: 16043169
doi: 10.1016/j.cognition.2005.05.006
Rieser, J. J., Lockman, J. J. & Pick, H. L. The role of visual experience in knowledge of spatial layout. Percept. Psychophys. 28, 185–190 (1980).
pubmed: 7432996
doi: 10.3758/BF03204374
Gilmartin, P. P. Maps, mental imagery, and gender in the recall of geographical information. Am. Cartograph. 13, 335–344 (1986).
doi: 10.1559/152304086783887226
Velez, M., Silver, D. & Tremaine, M. Understanding visualization through spatial ability differences. in VIS 05. IEEE Visualization, 2005., 511–518. https://doi.org/10.1109/VISUAL.2005.1532836 (2005).
Kosslyn, S. M. Image and Mind (Harvard University Press, 1980).
Kosslyn, S. M. Mental images and the brain. Cognit. Neuropsychol. 22, 333–347 (2005).
doi: 10.1080/02643290442000130
Vecchi, T. & Cornoldi, C. Passive storage and active manipulation in visuo-spatial working memory: Further evidence from the study of age differences. Eur. J. Cognit. Psychol. 11, 391–406 (1999).
doi: 10.1080/713752324
Nori, R. & Piccardi, L. Familiarity and spatial cognitive style: How important are they for spatial representation. In Spatial Memory: Visuospatial Processes, Cognitive Performance and Developmental Effects . (Ed. Thomas, J. B.) 123–144 (Nova Science Publishers, Inc., 2011).
Schank, R. C., Berman, T. R. & Macpherson, K. A. Learning by doing. Instruct.-Design Theories Models New Paradigm Instruct. Theory 2, 161–181 (1999).
Leo, F. et al. Improving spatial working memory in blind and sighted youngsters using programmable tactile displays. SAGE Open Med. 6, 2050312118820028 (2018).
pubmed: 30574309
pmcid: 6299321
doi: 10.1177/2050312118820028
Rindermann, H., Ackermann, A. L. & Te Nijenhuis, J. Does blindness boost working memory? A natural experiment and cross-cultural study. Front. Psychol. 11, 1571 (2020).
pubmed: 32719644
pmcid: 7347789
doi: 10.3389/fpsyg.2020.01571
Baddeley, A. Working memory. Curr. Biol. 20, R136–R140. https://doi.org/10.1016/j.cub.2009.12.014 (2010).
doi: 10.1016/j.cub.2009.12.014
pubmed: 20178752
Setti, W., Cuturi, L. F., Engel, I., Picinali, L. & Gori, M. The influence of early visual deprivation on audio-spatial working memory. Neuropsychology 36, 55(2021).
Vecchi, T., Tinti, C. & Cornoldi, C. Spatial memory and integration processes in congenital blindness. Neuroreport 15, 2787–2790 (2004).
pubmed: 15597055
Di Nuovo, S. F., Angelica, A., Santoro, G. & Platania, S. Intelligence and mental imagery in intellectual disability. Mediterranean J. Clin. Psychol. 6(2), (2018).
Uttal, D. H. et al. The malleability of spatial skills: A meta-analysis of training studies. Psychol. Bull. 139, 352 (2013).
pubmed: 22663761
doi: 10.1037/a0028446
Blades, M., Lippa, Y., Golledge, R. G., Jacobson, R. D. & Kitchin, R. M. The effect of spatial tasks on visually impaired peoples’ wayfinding abilities. J. Visual Impairment Blindness 96, 407–419 (2002).
doi: 10.1177/0145482X0209600604
Almeida, M. D. X., Martins, L. B. & Lima, F. J. Analysis of wayfinding strategies of blind people using tactile maps. Procedia Manufact. 3, 6020–6027 (2015).
doi: 10.1016/j.promfg.2015.07.716
Palivcová, D., Macík, M. & Míkovec, Z. Interactive tactile map as a tool for building spatial knowledge of visually impaired older adults. in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 1–9 (2020).
Espinosa, M. & Ochaita, E. Using tactile maps to improve the practical spatial knowledge of adults who are blind. J. Visual Impairment Blindness 92, 338–345 (1998).
doi: 10.1177/0145482X9809200512
Jacobson, R. D. Cognitive mapping without sight: Four preliminary studies of spatial learning. J. Environ. Psychol. 18, 289–305 (1998).
doi: 10.1006/jevp.1998.0098
Brock, A. M., Truillet, P., Oriola, B., Picard, D. & Jouffrais, C. Interactivity improves usability of geographic maps for visually impaired people. Hum.-Comput. Interact. 30, 156–194 (2015).
doi: 10.1080/07370024.2014.924412
Shi, L., Zhao, Y., Gonzalez Penuela, R., Kupferstein, E. & Azenkot, S. Molder: an accessible design tool for tactile maps. in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14 (2020).
Landau, S. & Wells, L. Merging tactile sensory input and audio data by means of the talking tactile tablet. in Proc. Eurohaptics, vol. 3 (2003).
Reynolds, Z. & Glenney, B. When sensory substitution devices strike back: An interactive training paradigm. Philos. Study 2, 432 (2012).
Jansson, G. Can a haptic display rendering of virtual three-dimensional objects be useful for people with visual impairments?. J. Visual Impairment Blindness 93, 426–429 (1999).
doi: 10.1177/0145482X9909300707
Jansen-Osmann, P. Using desktop virtual environments to investigate the role of landmarks. Comput. Hum. Behav. 18, 427–436 (2002).
doi: 10.1016/S0747-5632(01)00055-3
Connors, E. C., Chrastil, E. R., Sánchez, J. & Merabet, L. B. Action video game play and transfer of navigation and spatial cognition skills in adolescents who are blind. Front. Hum. Neurosci. 8, 133 (2014).
pubmed: 24653690
pmcid: 3949101
doi: 10.3389/fnhum.2014.00133
Merabet, L., Connors, E., Halko, M. & Sánchez, J. Teaching the blind to find their way by playing video games. PLoS ONE 7, e44958 (2012).
pubmed: 23028703
pmcid: 3446956
doi: 10.1371/journal.pone.0044958
Katz, B. F. et al. Navig: Augmented reality guidance system for the visually impaired: Combining object localization, gnss, and spatial audio. Virtual Reality 16, 253–269 (2012).
doi: 10.1007/s10055-012-0213-6
Ziat, M., Gapenne, O., Stewart, J., Lenay, C. & Bausse, J. Design of a haptic zoom: levels and steps. in Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07), 102–108 (IEEE, 2007).
Rastogi, R., Pawluk, T. D. & Ketchum, J. Intuitive tactile zooming for graphics accessed by individuals who are blind and visually impaired. IEEE Trans. Neural Syst. Rehabilit. Eng. 21, 655–663 (2013).
doi: 10.1109/TNSRE.2013.2250520
Prescher, D. & Weber, G. Comparing two approaches of tactile zooming on a large pin-matrix device. in IFIP Conference on Human-Computer Interaction, 173–186 (Springer, 2017).
Gutiérrez-Fernández, A., Fernández-Llamas, C., Esteban, G. & Conde, M. Á. Haptic zoom: An interaction model for desktop haptic devices with limited workspace. Int. J. Hum.–Comput. Interact. 1–12 (2022).
Lahav, O. Virtual reality systems as an orientation aid for people who are blind to acquire new spatial information. Sensors 22, 1307 (2022).
pubmed: 35214209
pmcid: 8963089
doi: 10.3390/s22041307
Lahav, O. & Mioduser, D. Blind persons’ acquisition of spatial cognitive mapping and orientation skills supported by virtual environment. Int. J. Disability Hum. Develop. 4, 231 (2005).
doi: 10.1515/IJDHD.2005.4.3.231
Lahav, O. & Mioduser, D. Haptic-feedback support for cognitive mapping of unknown spaces by people who are blind. Int. J. Hum.-Comput. Stud. 66, 23–35 (2008).
doi: 10.1016/j.ijhcs.2007.08.001
Schinazi, V. R., Thrash, T. & Chebat, D.-R. Spatial navigation by congenitally blind individuals. Wiley Interdiscip. Rev. Cognit. Sci. 7, 37–58 (2016).
doi: 10.1002/wcs.1375
Hersh, M. A. & Johnson, M. A. Assistive Technology for Visually Impaired and Blind People Vol. 1 (Springer, 2008).
doi: 10.1007/978-1-84628-867-8
Geruschat, D. R., Turano, K. A. & Stahl, J. W. Traditional measures of mobility performance and retinitis pigmentosa. Optometry Vision Sci. 75, 525–537 (1998).
doi: 10.1097/00006324-199807000-00022
Thinus-Blanc, C. & Gaunet, F. Representation of space in blind persons: Vision as a spatial sense?. Psychol. Bull. 121, 20 (1997).
pubmed: 9064698
doi: 10.1037/0033-2909.121.1.20
Yazzolino, L. A., Connors, E. C., Hirsch, G. V., Sánchez, J. & Merabet, L. B. Developing virtual environments for learning and enhancing skills for the blind: Incorporating user-centered and neuroscience based approaches. In Virtual Reality for Psychological and Neurocognitive Interventions. 361–385 (Springer, 2019).
Giudice, N. A. Navigating without vision: principles of blind spatial cognition (2018).
Goldish, L. H. & Taylor, H. E. The optacon: A valuable device for blind persons. New Outlook Blind 68, 49–56 (1974).
McIntyre, S., Seizova-Cajic, T., Birznieks, I., Holcombe, A. O. & Vickery, R. M. Adaptation to motion presented with a tactile array. in International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, 351–359 (Springer, 2014).
Goncu, C. & Marriott, K. Gravvitas: Generic multi-touch presentation of accessible graphics. in IFIP Conference on Human-Computer Interaction, 30–48 (Springer, 2011).
Lévesque, V., Pasquero, J., Hayward, V. & Legault, M. Display of virtual braille dots by lateral skin deformation: Feasibility study. ACM Trans. Appl. Percept. (TAP) 2, 132–149 (2005).
doi: 10.1145/1060581.1060587
Rastogi, R., Pawluk, D. T. & Ketchum, J. M. Issues of using tactile mice by individuals who are blind and visually impaired. IEEE Trans. Neural Syst. Rehabilit. Eng. 18, 311–318 (2010).
doi: 10.1109/TNSRE.2010.2045008
White, G. R., Fitzpatrick, G. & McAllister, G. Toward accessible 3d virtual environments for the blind and visually impaired. in Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts, 134–141 (ACM, 2008).
de Pascale, M., Mulatto, S. & Prattichizzo, D. Bringing haptics to second life. In Proceedings of the 2008 Ambi-Sys workshop on Haptic user interfaces in ambient media systems, 6 (ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2008).
Ullrich, S., Knott, T., Law, Y. C., Grottke, O. & Kuhlen, T. Influence of the bimanual frame of reference with haptics for unimanual interaction tasks in virtual environments. in 2011 IEEE Symposium on 3D User Interfaces (3DUI), 39–46. https://doi.org/10.1109/3DUI.2011.5759215 (2011).
Garcia Robledo, P., Ortego, J., Ferre, M., Barrio, J. & Sanchez-Uran, M. A. Segmentation of bimanual virtual object manipulation tasks using multifinger haptic interfaces. IEEE Trans. Instrument. Meas. 60, 69–80. https://doi.org/10.1109/TIM.2010.2065690 (2011).
Hinckley, K., Pausch, R., Proffitt, D., Patten, J. & Kassell, N. Cooperative bimanual action. in Proceedings of the ACM SIGCHI Conference on Human factors in Computing Systems, 27–34 (1997).
Nanjappan, V. et al. User elicited dual-hand interactions for manipulating 3d objects in virtual reality environments. Human-Centric Comput. Inform. Sci. 8, 1–16 (2018).
Squeri, V. et al. Two hands, one perception: How bimanual haptic information is combined by the brain. J. Neurophysiol. 107, 544–550 (2012).
pubmed: 22031771
doi: 10.1152/jn.00756.2010
Hinckley, K., Pausch, R. & Proffitt, D. Attention and visual feedback: The bimanual frame of reference. in Proceedings of the 1997 symposium on Interactive 3D graphics, 121–ff (1997).
Buxton, W. & Myers, B. A study in two-handed input. ACM SIGCHI Bull. 17, 321–326 (1986).
doi: 10.1145/22339.22390
Guiard, Y. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. J. Motor Behav. 19, 486–517 (1987).
doi: 10.1080/00222895.1987.10735426
Talvas, A. Bimanual Haptic Interaction with Virtual Environments. Ph.D. thesis, INSA de Rennes (2014).
Stone, K. D., Bryant, D. C. & Gonzalez, C. L. Hand use for grasping in a bimanual task: Evidence for different roles?. Exp. Brain Res. 224, 455–467 (2013).
pubmed: 23161156
doi: 10.1007/s00221-012-3325-z
Brooke, J. et al. Sus-a quick and dirty usability scale. Usability Eval. Ind. 189, 4–7 (1996).
Bangor, A., Kortum, P. T. & Miller, J. T. An empirical evaluation of the system usability scale. Intl. J. Hum.-Comput. Interact. 24, 574–594 (2008).
doi: 10.1080/10447310802205776
Coluccia, E. & Louse, G. Gender differences in spatial orientation: A review. J. Environ. Psychol. 24, 329–340 (2004).
doi: 10.1016/j.jenvp.2004.08.006
Brayda, L., Campus, C., Memeo, M. & Lucagrossi, L. The importance of visual experience, gender and emotion in the assessment of an assistive tactile mouse. IEEE Trans. Haptics. https://doi.org/10.1109/TOH.2015.2426692 (2015).
doi: 10.1109/TOH.2015.2426692
pubmed: 25935047
Sandstrom, N. J., Kaufman, J. & Huettel, S. A. Males and females use different distal cues in a virtual environment navigation task. Cognit. Brain Res. 6, 351–360 (1998).
doi: 10.1016/S0926-6410(98)00002-0
Dabbs, J. M. Jr., Chang, E.-L., Strong, R. A. & Milun, R. Spatial ability, navigation strategy, and geographic knowledge among men and women. Evolut. Hum. Behav. 19, 89–98 (1998).
doi: 10.1016/S1090-5138(97)00107-4
Vélaz, Y., Lozano-Rodero, A., Suescun, A. & Gutiérrez, T. Natural and hybrid bimanual interaction for virtual assembly tasks. Virtual Reality 18, 161–171 (2014).
doi: 10.1007/s10055-013-0240-y
Vyawahare, V. & Vance, J. Human centered multimodal 3d user interface for desktop vr assembly. in Proceedings of the Emerging Technologies Conference (2009).
Kantowitz, B. H. Effects of response symmetry upon bi-manual rapid aiming. in Proceedings of the Human Factors Society Annual Meeting, vol. 35, 1541–1545 (SAGE Publications Sage CA: Los Angeles, CA, 1991).
Talvas, A., Marchal, M. & Lécuyer, A. A survey on bimanual haptic interaction. IEEE Trans. Haptics 7, 285–300 (2014).
pubmed: 25248213
doi: 10.1109/TOH.2014.2314456
Balakrishnan, R. & Hinckley, K. Symmetric bimanual interaction. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 33–40 (2000).
Ulinski, A., Zanbaka, C., Wartell, Z., Goolkasian, P. & Hodges, L. F. Two handed selection techniques for volumetric data. in 2007 IEEE Symposium on 3D User Interfaces (IEEE, 2007).
Kabbash, P., Buxton, W. & Sellen, A. Two-handed input in a compound task. in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 417–423 (1994).
Memeo, M., Jacono, M., Sandini, G. & Brayda, L. Enabling visually impaired people to learn three-dimensional tactile graphics with a 3dof haptic mouse. J. NeuroEng. Rehabilit. 18, 1–21 (2021).
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2013).
Royston, P. Remark as r94: A remark on algorithm as 181: The w-test for normality. J. R. Stat. Soc. Series C (Appl. Stat.). 44, 547–551 (1995).
Benjamini, Y. & Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29(4), 1165–1188 (JSTOR, 2001).