Role of reinforcement learning for risk-based robust control of cyber-physical energy systems.
cyber-physical energy systems
reinforcement learning
risk analysis
robust control
Journal
Risk analysis : an official publication of the Society for Risk Analysis
ISSN: 1539-6924
Titre abrégé: Risk Anal
Pays: United States
ID NLM: 8109978
Informations de publication
Date de publication:
Nov 2023
Nov 2023
Historique:
revised:
14
10
2022
received:
15
07
2021
accepted:
27
12
2022
medline:
7
2
2023
pubmed:
7
2
2023
entrez:
6
2
2023
Statut:
ppublish
Résumé
Critical infrastructures such as cyber-physical energy systems (CPS-E) integrate information flow and physical operations that are vulnerable to natural and targeted failures. Safe, secure, and reliable operation and control of CPS-E is critical to ensure societal well-being and economic prosperity. Automated control is key for real-time operations and may be mathematically cast as a sequential decision-making problem under uncertainty. Emergence of data-driven techniques for decision making under uncertainty, such as reinforcement learning (RL), have led to promising advances for addressing sequential decision-making problems for risk-based robust CPS-E control. However, existing research challenges include understanding the applicability of RL methods across diverse CPS-E applications, addressing the effect of risk preferences across multiple RL methods, and development of open-source domain-aware simulation environments for RL experimentation within a CPS-E context. This article systematically analyzes the applicability of four types of RL methods (model-free, model-based, hybrid model-free and model-based, and hierarchical) for risk-based robust CPS-E control. Problem features and solution stability for the RL methods are also discussed. We demonstrate and compare the performance of multiple RL methods under different risk specifications (risk-averse, risk-neutral, and risk-seeking) through the development and application of an open-source simulation environment. Motivating numerical simulation examples include representative single-zone and multizone building control use cases. Finally, six key insights for future research and broader adoption of RL methods are identified, with specific emphasis on problem features, algorithmic explainability, and solution stability.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
2280-2297Subventions
Organisme : U.S. Department of Energy
Organisme : Pacific Northwest National Laboratory
Informations de copyright
© 2023 Battelle Memorial Institute. Risk Analysis published by Wiley Periodicals LLC on behalf of Society for Risk Analysis.
Références
Alur, R. (2015). Principles of cyber-physical systems. MIT Press.
Ashibani, Y., & Mahmoud, Q. H. (2017). Cyber physical systems security: Analysis, challenges and solutions. Computers & Security, 68, 81-97.
Audigier, M. A., Kiremidjiam, A. S., Chiu, S. S., & King, S. A. (2000). Risk analysis of port facilities. In Proceedings of 12th World Conference on Earthquake Engineering, Paper No. 2311.
Baggott, S., & Santos, J. (2020). A risk analysis framework for cyber security and critical infrastructure protection of the US electric power grid. Risk Analysis, 40(9), 1744-1761.
Baker, J., & Cornell, C. A. (2006). Vector-valued ground motion intensity measures for probabilistic seismic demand analysis. Pacific Earthquake Engineering Research (PEER) Report 2006/08, PEER Center, University of California-Berkeley, 368 pp.
Barto, A. G., & Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(1-2), 41-77.
Berseth, G., Kyriazis, A., Zinin, I., Choi, W., & van de Panne, M. (2018). Model-based action exploration for learning dynamic motion skills. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1540-1546. IEEE.
Bhattacharya, A., Bopardikar, S., Chatterjee, S., & Vrabie, D. (2019). Cyber threat screening using a queuing-based game theoretic approach. Journal of Information Warfare, 18(4), 37-52.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). OpenAI Gym. arXiv preprint. https://doi.org/10.48550/arXiv.1606.01540
Cardenas, A., Amin, S., Sinopoli, B., Giani, A., Perrig, A., & Sastry, S. (2009). Challenges for securing cyber physical systems. In Workshop on future directions in cyber-physical systems security (Vol., 5, No. 1). DHS.
Chatterjee, S., Brigantic, R. T., & Waterworth, A. M. (2021). An overview of risk modeling methods and approaches for national security. In Chatterjee, S., Brigantic, R. T., & Waterworth, A. M. (Eds.). Applied risk analysis for guiding homeland security policy and decisions. (pp. 69-100). Wiley.
Che, T., Lu, Y., Tucker, G., Bhupatiraju, S., Gu, S., Levine, S., & Bengio, Y. (2018). Combining model-based and model-free RL via multi-step control variates. https://openreview.net/forum?id=HkPCrEZ0Z
Cornell, C. A., & Krawinkler, H. (2000). Progress and challenges in seismic performance assessment. PEER Center News, 3(2), 1-3.
Dong, L., Li, Y., Zhou, X., Wen, Y., & Guan, K. (2020). Intelligent trainer for dyna-style model-based deep reinforcement learning. IEEE Trans Neural Networks and Learn Systems. IEEE.
Du, Y., Zandi, H., Kotevska, O., Kurte, K., Munk, J., Amasyali, K., Mckee, E., & Li, F. (2021). Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning. Applied Energy, 281, 116-117.
Dulac-Arnold, G., Mankowitz, D., & Hester, T. (2019). Challenges of real-world reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, PMLR 97, 1-13.
Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., & Levine, S. (2018). Model-based value expansion for efficient model-free reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018).
Garrick, B. J. (2008). Quantifying and controlling catastrophic risks. Academic Press.
Günay, S., & Mosalam, K. M. (2013). PEER performance-based earthquake engineering methodology, revisited. Journal of Earthquake Engineering, 17, 829-858.
Han, M., Tian, Y., Zhang, L., Wang, J., & Pan, W. (2019). Model-free control of nonlinear stochastic systems with stability guarantee. https://openreview.net/pdf?id=SkeWc2EKPH
Haydari, A., & Yilmaz, Y. (2020). Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 1-22.
Heracleous, C., Kolios, P., Panayiotou, C. G., Ellinas, G., & Polycarpou, M. M. (2017). Hybrid systems modeling for critical infrastructures interdependency analysis. Reliability Engineering & System Safety, 165, 89-101.
Huang, Q., Huang, R., Hao, W., Tan, J., Fan, R., & Huang, Z. (2020). Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid, 11(2), 1171-1182.
JetBrains. (2022). PyCharm. [online] JetBrains. https://www.jetbrains.com/pycharm/
Jiang, D. R., & Powell, W. B. (2015). Optimal hour-ahead bidding in the real-time electricity market with battery storage using approximate dynamic programming. INFORMS Journal on Computing, 27(3), 525-543.
Kaplan, S., & Garrick, B. J. (1981). On the quantitative definition of risk. Risk Analysis, 1(1), 11-27.
Kim, K. D., & Kumar, P. R. (2013). An overview and some challenges in cyber-physical systems. Journal of the Indian Institute of Science, 93(3), 341-352.
Kim, T., & Kim, H. J. (2016). Path tracking control and identification of tire parameters using on-line model-based reinforcement learning. In 2016 16th International Conference on Control, Automation and Systems (ICCAS), (pp. 215-219).
Kochenderfer, M. J. (2015). Decision making under uncertainty: Theory and application. MIT Press.
Liu, J., Liu, X., Koo, T - K J., Sinopoli, B., Sastry, S., & Lee, E. (1999). A hierarchical hybrid system model and its simulation. In Proceedings of the 38th IEEE conference on decision and control, 4, 3508-13. http://doi.org/10.1109/CDC.1999.827883
Lu, N. (2012). An evaluation of the HVAC load potential for providing load balancing service. IEEE Transactions on Smart Grid, 3(3), 1263-1270.
Lygeros, J., Tomlin, C., & Sastry, S. (2008). Hybrid systems: Modeling, analysis and control. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.461.7873&rep=rep1&type=pdf
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., & Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.
Moehle, J., & Deierlein, G G. (2004) A framework methodology for performance-based earthquake engineering. In Proceedings of 13th World Conference on Earthquake Engineering, (Vol. 679). Vancouver, Canada, 13 pp.
Moerland, T. M., Broekens, J., & Jonker, C. M. (2020). Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006.16712.
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018). Data-efficient hierarchical reinforcement learning. In Proceedings of the 31st Advances in Neural Information Processing Systems (NeurIPS 2018).
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018). Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA). (pp. 7559-7566). IEEE.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., & Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd Advances in Neural Information Processing Systems (NeurIPS 2019). (pp. 8024-8035.
Porter, K. A. (2003). An overview of PEER's performance-based earthquake engineering methodology. In Proceedings of 9th International Conference on Application of Statistics and Probability in Civil Engineering, San Francisco, California, 8 pp.
Rao, N. S., Poole, S. W., Ma, C. Y., He, F., Zhuang, J., & Yau, D. K. (2016). Defense of cyber infrastructures against cyber-physical attacks using game-theoretic models. Risk Analysis, 36(4), 694-710.
Russell, S. J., & Norvig, P. (2016). Artificial intelligence: A modern approach. Pearson Education Limited.
Schmidt, M., & Åhlund, C. (2018). Smart buildings as cyber-physical systems: Data-driven predictive control strategies for energy efficiency. Renewable and Sustainable Energy Reviews, 90, 742-756.
Shen, Y., Tobia, M. J., Sommer, T., & Obermayer, K. (2014). Risk-sensitive reinforcement learning. Neural Computation, 26(7), 1298-1328.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484-489.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., & Lillicrap, T. (2018). A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419), 1140-1144.
Sun, C. C., Liu, C. C., & Xie, J. (2016). Cyber-physical system security of a power grid: State-of-the-art. Electronics, 5(3), 40.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
Wang, W., Di Maio, F., & Zio, E. (2019). Adversarial risk analysis to allocate optimal defense resources for protecting cyber-physical systems from cyber attacks. Risk Analysis, 39(12), 2766-2785.
Wang, X., Xiong, W., Wang, H., & Wang, W. (2018). Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In Ferrari, V., Hebert, M., Sminchisescu, C., & Weiss, Y. (Eds). Computer vision - ECCV 2018. ( Vol. 11220, pp. 37-53). Springer.
Xiao, Q., Cao, Z., & Zhou, M. (2019). Learning locomotion skills via model-based proximal meta-reinforcement learning. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), (pp. 1545-1550). IEEE.
Yu, L., Sun, Y., Xu, Z., Shen, C., Yue, D., Jiang, T., & Guan, X. (2020). Multi-agent deep reinforcement learning for HVAC control in commercial buildings. IEEE Transactions on Smart Grid, 12(1), 407-419.