Outracing champion Gran Turismo drivers with deep reinforcement learning.
Journal
Nature
ISSN: 1476-4687
Titre abrégé: Nature
Pays: England
ID NLM: 0410462
Informations de publication
Date de publication:
02 2022
02 2022
Historique:
received:
09
08
2021
accepted:
15
12
2021
entrez:
10
2
2022
pubmed:
11
2
2022
medline:
16
4
2022
Statut:
ppublish
Résumé
Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits
Identifiants
pubmed: 35140384
doi: 10.1038/s41586-021-04357-7
pii: 10.1038/s41586-021-04357-7
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
223-228Commentaires et corrections
Type : CommentIn
Informations de copyright
© 2022. The Author(s), under exclusive licence to Springer Nature Limited.
Références
Milliken, W. F. et al. Race Car Vehicle Dynamics Vol. 400 (Society of Automotive Engineers, 1995).
Mnih, V. et al. Playing Atari with deep reinforcement learning. Preprint at https://arxiv.org/abs/1312.5602 (2013).
Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
doi: 10.1038/nature16961
Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
doi: 10.1038/nature24270
Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).
doi: 10.1038/s41586-019-1724-z
Berner, C. et al. Dota 2 with large scale deep reinforcement learning. Preprint at https://arxiv.org/abs/1912.06680 (2019).
Laurense, V. A., Goh, J. Y. & Gerdes, J. C. In 2017 American Control Conference (ACC) 5586–5591 (IEEE, 2017).
Spielberg, N. A., Brown, M., Kapania, N. R., Kegelman, J. C. & Gerdes, J. C. Neural network vehicle models for high-performance automated driving. Sci. Robot. 4, eaaw1975 (2019).
doi: 10.1126/scirobotics.aaw1975
Burke, K. Data makes it beta: Roborace returns for second season with updateable self-driving vehicles powered by NVIDIA DRIVE. The Official NVIDIA Blog https://blogs.nvidia.com/blog/2020/10/29/roborace-second-season-nvidia-drive/ (2020).
Leporati, G. No driver? no problem—this is the Indy Autonomous Challenge. Ars Technica https://arstechnica.com/cars/2021/07/a-science-fair-or-the-future-of-racing-the-indy-autonomous-challenge/ (2021).
Williams, G., Drews, P., Goldfain, B., Rehg, J. M. & Theodorou, E. A. In 2016 IEEE International Conference on Robotics and Automation (ICRA) 1433–1440 (IEEE, 2016).
Williams, G., Drews, P., Goldfain, B., Rehg, J. M. & Theodorou, E. A. Information-theoretic model predictive control: theory and applications to autonomous driving. IEEE Trans. Robot. 34, 1603–1622 (2018).
doi: 10.1109/TRO.2018.2865891
Pan, Y. et al. In Proc. Robotics: Science and Systems XIV (eds Kress-Gazit, H., Srinivasa, S., Howard, T. & Atanasov, N.) https://doi.org/10.15607/RSS.2018.XIV.056 (Carnegie Mellon Univ., 2018).
Pan, Y. et al. Imitation learning for agile autonomous driving. Int. J. Robot. Res. 39, 286–302 (2020).
doi: 10.1177/0278364919880273
Amazon Web Services. AWS DeepRacer League. https://aws.amazon.com/deepracer/league/ (2019).
Pyeatt, L. D. & Howe, A. E. Learning to race: experiments with a simulated race car. In Proc. Eleventh International FLAIRS Conference 357–361 (AAAI, 1998).
Chaperot, B. & Fyfe, C. In 2006 IEEE Symposium on Computational Intelligence and Games 181–186 (IEEE, 2006).
Cardamone, L., Loiacono, D. & Lanzi, P. L. In Proc. 11th Annual Conference on Genetic and Evolutionary Computation 1179–1186 (ACM, 2009).
Cardamone, L., Loiacono, D. & Lanzi, P. L. In 2009 IEEE Congress on Evolutionary Computation 2622–2629 (IEEE, 2009).
Loiacono, D., Prete, A., Lanzi, L. & Cardamone, L. In IEEE Congress on Evolutionary Computation 1–8 (IEEE, 2010).
Jaritz, M., de Charette, R., Toromanoff, M., Perot, E. & Nashashibi, F. In 2018 IEEE International Conference on Robotics and Automation (ICRA) 2070–2075 (IEEE, 2018).
Weiss, T. & Behl, M. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE) 1163–1168 (IEEE, 2020).
Weiss, T., Babu, V. S. & Behl, M. In NeurIPS 2020 Workshop on Machine Learning for Autonomous Driving (NeurIPS, 2020).
Fuchs, F., Song, Y., Kaufmann, E., Scaramuzza, D. & Dürr, P. Super-human performance in Gran Turismo Sport using deep reinforcement learning. IEEE Robot. Autom. Lett. 6, 4257–4264 (2021).
doi: 10.1109/LRA.2021.3064284
Song, Y., Lin, H., Kaufmann, E., Dürr, P. & Scaramuzza, D. In Proc. IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2021).
Theodosis, P. A. & Gerdes, J. C. In Dynamic Systems and Control Conference Vol. 45295, 235–241 (American Society of Mechanical Engineers, 2012).
Funke, J. et al. In 2012 IEEE Intelligent Vehicles Symposium 541–547 (IEEE, 2012).
Kritayakirana, K. & Gerdes, J. C. Autonomous vehicle control at the limits of handling. Int. J. Veh. Auton. Syst. 10, 271–296 (2012).
doi: 10.1504/IJVAS.2012.051270
Bonkowski, J. Here’s what you missed from the Indy Autonomous Challenge main event. Autoweek https://www.autoweek.com/racing/more-racing/a38069263/what-missed-indy-autonomous-challenge-main-event/ (2021).
Rutherford, S. J. & Cole, D. J. Modelling nonlinear vehicle dynamics with neural networks. Int. J. Veh. Des. 53, 260–287 (2010).
doi: 10.1504/IJVD.2010.034101
Pomerleau, D. A. In Robot Learning (eds Connell, J. H. & Mahadevan, S.) 19–43 (Springer, 1993).
Togelius, J. & Lucas, S. M. In 2006 IEEE International Conference on Evolutionary Computation 1187–1194 (IEEE, 2006).
Schwarting, W. et al. Deep latent competition: learning to race using visual control policies in latent space. Preprint at https://arxiv.org/abs/2102.09812 (2021).
Gozli, D. G., Bavelier, D. & Pratt, J. The effect of action video game playing on sensorimotor learning: evidence from a movement tracking task. Hum. Mov. Sci. 38, 152–162 (2014).
doi: 10.1016/j.humov.2014.09.004
Davids, K., Williams, A. M. & Williams, J. G. Visual Perception and Action in Sport (Routledge, 2005).
Haarnoja, T., Zhou, A., Abbeel, P. & Levine, S. In Proc. 35th International Conference on Machine Learning 1856–1865 (PMLR, 2018).
Haarnoja, T. et al. Soft actor-critic algorithms and applications. Preprint at https://arxiv.org/abs/1812.05905 (2018).
Mnih, V. et al. In Proc. 33rd International Conference on Machine Learning 1928–1937 (PMLR, 2016).
Dabney, W., Rowland, M., Bellemare, M. G. & Munos, R. In 32nd AAAI Conference on Artificial Intelligence (AAAI, 2018).
Lin, L.-J. Reinforcement Learning for Robots Using Neural Networks. Dissertation, Carnegie Mellon Univ. (1993).
Siu, H. C. et al. Evaluation of human-AI teams for learned and rule-based agents in Hanabi. Preprint at https://arxiv.org/abs/2107.07630 (2021).
Tesauro, G. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6, 215–219 (1994).
doi: 10.1162/neco.1994.6.2.215
Devore, J. L. Probability and Statistics for Engineering and the Sciences 6th edn (Brooks/Cole, 2004).
Xia, L., Zhou, Z., Yang, J. & Zhao, Q. DSAC: distributional soft actor critic for risk-sensitive reinforcement learning. Preprint at https://arxiv.org/abs/2004.14547 (2020).
Fujimoto, S., van Hoof, H. & Meger, D. In Proc. 35th International Conference on Machine Learning 1587–1596 (PMLR, 2018).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
Liu, Z., Li, X., Kang, B. & Darrell, T. In International Conference on Learning Representations (ICLR, 2021).
Kingma, D. P. & Ba, J. In International Conference on Learning Representations (ICLR, 2015).
Cassirer, A. et al. Reverb: a framework for experience replay. Preprint at https://arxiv.org/abs/2102.04736 (2021).
Narvekar, S., Sinapov, J., Leonetti, M. & Stone, P. In Proc. 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016) (2016).