Reinforcement learning-trained optimisers and Bayesian optimisation for online particle accelerator tuning.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
08 Jul 2024
08 Jul 2024
Historique:
received:
06
12
2023
accepted:
01
07
2024
medline:
9
7
2024
pubmed:
9
7
2024
entrez:
8
7
2024
Statut:
epublish
Résumé
Online tuning of particle accelerators is a complex optimisation problem that continues to require manual intervention by experienced human operators. Autonomous tuning is a rapidly expanding field of research, where learning-based methods like Bayesian optimisation (BO) hold great promise in improving plant performance and reducing tuning times. At the same time, reinforcement learning (RL) is a capable method of learning intelligent controllers, and recent work shows that RL can also be used to train domain-specialised optimisers in so-called reinforcement learning-trained optimisation (RLO). In parallel efforts, both algorithms have found successful adoption in particle accelerator tuning. Here we present a comparative case study, assessing the performance of both algorithms while providing a nuanced analysis of the merits and the practical challenges involved in deploying them to real-world facilities. Our results will help practitioners choose a suitable learning-based tuning algorithm for their tuning tasks, accelerating the adoption of autonomous tuning algorithms, ultimately improving the availability of particle accelerators and pushing their operational limits.
Identifiants
pubmed: 38977749
doi: 10.1038/s41598-024-66263-y
pii: 10.1038/s41598-024-66263-y
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
15733Subventions
Organisme : Helmholtz-Gemeinschaft
ID : InternLabs-0011
Organisme : Helmholtz Artificial Intelligence Cooperation Unit
ID : ZT-I-PF-5-6
Informations de copyright
© 2024. The Author(s).
Références
Huang, X. Beam-based correction and optimization for accelerators (Taylor & Francis, 2020).
Bergan, W. F. et al. Online storage ring optimization using dimension-reduction and genetic algorithms. Phys. Rev. Acceler. Beams 22, 054601. https://doi.org/10.1103/PhysRevAccelBeams.22.054601 (2019).
doi: 10.1103/PhysRevAccelBeams.22.054601
Huang, X., Corbett, J., Safranek, J. & Wu, J. An algorithm for online optimization of accelerators. Nucl. Instrum. Methods Phys. Res., Sect. A 726, 77–83. https://doi.org/10.1016/j.nima.2013.05.046 (2013).
doi: 10.1016/j.nima.2013.05.046
Bellman, R. Dynamic Programming (Princeton University Press, 1957).
Roussel, R. et al. Bayesian optimization algorithms for accelerator physics (2024). arXiv:2312.05667 .
Duris, J. et al. Bayesian Optimization of a Free-Electron Laser. Phys. Rev. Lett.124 (2020).
Hanuka, A. et al. Online tuning and light source control using a physics-informed Gaussian process. In Proceedings of the 33rd Conference on Neural Information Processing Systems (2019).
Jalas, S. et al. Bayesian optimization of a laser-plasma accelerator. Phys. Rev. Lett. 126, 1. https://doi.org/10.1103/PhysRevLett.126.104801 (2021).
doi: 10.1103/PhysRevLett.126.104801
Roussel, R., Hanuka, A. & Edelen, A. Multiobjective bayesian optimization for online accelerator tuning. Phys. Rev. Accel. Beams 24, 062801. https://doi.org/10.1103/PhysRevAccelBeams.24.062801 (2021).
doi: 10.1103/PhysRevAccelBeams.24.062801
Xu, C. et al. Bayesian optimization of the beam injection process into a storage ring. Phys. Rev. Accel. Beams 26, 034601. https://doi.org/10.1103/PhysRevAccelBeams.26.034601 (2023).
doi: 10.1103/PhysRevAccelBeams.26.034601
McIntire, M., Cope, T., Ermon, S. & Ratner, D. Bayesian Optimization of FEL Performance at LCLS. In Proceedings of the 7th International Particle Accelerator Conference (2016).
Boltz, T. et al. Feedback design for control of the micro-bunching instability based on reinforcement learning. In CERN Yellow Reports: Conference Proceedings 9, 227–227 (2020).
St. John, J. et al. Real-time artificial intelligence for accelerator control: A study at the Fermilab Booster. Phys. Rev. Acceler. Beams 24, 104601 (2021).
Chen, T. et al. Learning to optimize: A primer and a benchmark. J. Mach. Learn. Res. 23, 1–59 (2022).
Li, K. & Malik, J. Learning to optimize. In International Conference on Learning Representations (2017).
Li, K. & Malik, J. Learning to optimize neural nets (2017). Preprint available at arXiv:1703.00441 .
Andrychowicz, M. et al. Learning to learn by gradient descent by gradient descent. In Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016) (2016).
Kain, V. et al. Sample-efficient reinforcement learning for CERN accelerator control. Phys. Rev. Acceler. Beams 23, 124801. https://doi.org/10.1103/PhysRevAccelBeams.23.124801 (2020).
doi: 10.1103/PhysRevAccelBeams.23.124801
Kaiser, J., Stein, O. & Eichler, A. Learning-based optimisation of particle accelerators under partial observability without real-world training. In Chaudhuri, K. et al. (eds.) Proceedings of the 39th International Conference on Machine Learning, vol. 162 of Proceedings of Machine Learning Research, 10575–10585 (PMLR, 2022).
Pang, X., Thulasidasan, S. & Rybarcyk, L. Autonomous control of a particle accelerator using deep reinforcement learning. In Proceedings of the Machine Learning for Engineering Modeling, Simulation, and Design Workshop at Neural Information Processing Systems 2020 (2020).
Meier, E., Biedron, S., LeBlanc, G., Morgan, M. & Wu, J. Electron beam energy and bunch length feed forward control studies using an artificial neural network at the linac coherent light source. Nucl. Instrum. Methods Phys. Res., Sect. A 610, 629–635. https://doi.org/10.1016/j.nima.2009.09.048 (2009).
doi: 10.1016/j.nima.2009.09.048
Edelen, A. L. et al. Neural networks for modeling and control of particle accelerators. IEEE Trans. Nuclear Sci. 63, 878–897. https://doi.org/10.1109/TNS.2016.2543203 (2016). Basically model predictive control accompanied by a nice literature overview.
Scheinker, A., Edelen, A., Bohler, D., Emma, C. & Lutman, A. Demonstration of model-independent control of the longitudinal phase space of electron beams in the linac-coherent light source with femtosecond resolution. Phys. Rev. Lett. 121, 044801. https://doi.org/10.1103/PhysRevLett.121.044801 (2018).
doi: 10.1103/PhysRevLett.121.044801
pubmed: 30095929
Degrave, J. et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602, 414–419. https://doi.org/10.1038/s41586-021-04301-9 (2022).
doi: 10.1038/s41586-021-04301-9
pubmed: 35173339
pmcid: 8850200
Seo, J. et al. Feedforward beta control in the KSTAR tokamak by deep reinforcement learning. Nuclear Fusion 61. https://doi.org/10.1088/1741-4326/ac121b (2021).
Seo, J. et al. Development of an operation trajectory design algorithm for control of multiple 0D parameters using deep reinforcement learning in KSTAR. Nuclear Fusion 62. https://doi.org/10.1088/1741-4326/ac79be (2022).
Guerra-Ramos, D., Trujillo-Sevilla, J. & Rodríguez-Ramos, J. M. Towards piston fine tuning of segmented mirrors through reinforcement learning. Appl. Sci. (Switzerland)10. https://doi.org/10.3390/app10093207 (2020).
Nousiainen, J. et al. Toward on-sky adaptive optics control using reinforcement learning: Model-based policy optimization for adaptive optics. Astron. Astrophysi. 664. https://doi.org/10.1051/0004-6361/202243311 (2022).
Yatawatta, S. & Avruch, I. M. Deep reinforcement learning for smart calibration of radio telescopes. Mon. Not. R. Astron. Soc. 505, 2141–2150. https://doi.org/10.1093/mnras/stab1401 (2021).
doi: 10.1093/mnras/stab1401
Zhou, Z., Li, X. & Zare, R. N. Optimizing chemical reactions with deep reinforcement learning. ACS Cent. Sci. 3, 1337–1344. https://doi.org/10.1021/acscentsci.7b00492 (2017).
doi: 10.1021/acscentsci.7b00492
pubmed: 29296675
pmcid: 5746857
Deneault, J. R. et al. Toward autonomous additive manufacturing: Bayesian optimization on a 3D printer. MRS Bull. 46, 566–575. https://doi.org/10.1557/s43577-021-00051-1 (2021).
doi: 10.1557/s43577-021-00051-1
Abdelrahman, H., Berkenkamp, F., Poland, J. & Krause, A. Bayesian optimization for maximum power point tracking in photovoltaic power plants. In 2016 European Control Conference (ECC), 2078–2083, https://doi.org/10.1109/ECC.2016.7810598 (Institute of Electrical and Electronics Engineers Inc., 2016).
Xiong, Y., Guo, L., Huang, Y. & Chen, L. Intelligent thermal control strategy based on reinforcement learning for space telescope. J. Thermophys. Heat Transfer 34, 37–44. https://doi.org/10.2514/1.T5774 (2020).
doi: 10.2514/1.T5774
Xiong, Y., Guo, L. & Tian, D. Application of deep reinforcement learning to thermal control of space telescope. J. Thermal Sci. Eng. Appl. 14. https://doi.org/10.1115/1.4051072 (2022).
Baheri, A., Bin-Karim, S., Bafandeh, A. & Vermillion, C. Real-time control using Bayesian optimization: A case study in airborne wind energy systems. Control. Eng. Pract. 69, 131–140. https://doi.org/10.1016/j.conengprac.2017.09.007 (2017).
doi: 10.1016/j.conengprac.2017.09.007
Maggi, L., Valcarce, A. & Hoydis, J. Bayesian optimization for radio resource management: Open loop power control. IEEE J. Sel. Areas Commun. 39, 1858–1871. https://doi.org/10.1109/JSAC.2021.3078490 (2021).
doi: 10.1109/JSAC.2021.3078490
Ding, X., Du, W. & Cerpa, A. E. MB2C: Model-based deep reinforcement learning for multi-zone building control. In BuildSys 2020 - Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, 50–59, https://doi.org/10.1145/3408308.3427986 (Association for Computing Machinery, Inc, 2020).
Nweye, K., Liu, B., Stone, P. & Nagy, Z. Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings. Energy AI 10. https://doi.org/10.1016/j.egyai.2022.100202 (2022).
Panofski, E. et al. Commissioning results and electron beam characterization with the S-band photoinjector at SINBAD-ARES. Instruments 5 (2021).
Burkart, F. et al. The ARES Linac at DESY. In Proceedings of the 31st International Linear Accelerator Conference (LINAC’22), no. 31 in International Linear Accelerator Conference, 691–694, https://doi.org/10.18429/JACoW-LINAC2022-THPOJO01 (JACoW Publishing, Geneva, Switzerland, 2022).
Rasmussen, C. E. & Williams, C. K. I. Gaussian Processes for Machine Learning (The MIT Press, 2005).
Nelder, J. A. & Mead, R. A simplex method for function minimization. Comput. J. 7 (1965).
Scheinker, A., Pang, X. & Rybarcyk, L. Model-independent particle accelerator tuning. Phys. Rev. ST Accel. Beams 16, 102803. https://doi.org/10.1103/PhysRevSTAB.16.102803 (2013).
doi: 10.1103/PhysRevSTAB.16.102803
Hwang, K. et al. Prior-mean-assisted bayesian optimization application on frib front-end tunning (2022). arXiv:2211.06400 .
Boltz, T. et al. More sample-efficient tuning of particle accelerators with bayesian optimization and prior mean models (2024). arXiv:2403.03225 .
Kaiser, J., Xu, C., Eichler, A. & Santamaria Garcia, A. Bridging the gap between machine learning and particle accelerator physics with high-speed, differentiable simulations. Phys. Rev. Acceler. Beams (2024).
Dulac-Arnold, G., Mankowitz, D. & Hester, T. Challenges of real-world reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning (2019).
Tobin, J. et al. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 23–30. https://doi.org/10.1109/IROS.2017.8202133 (2017).
OpenAI et al. Solving Rubik’s cube with a robot hand (2019). Preprint available at arXiv:1910.07113 .
Roussel, R. et al. Differentiable preisach modeling for characterization and optimization of particle accelerator systems with hysteresis. Phys. Rev. Lett. 128, 204801. https://doi.org/10.1103/PhysRevLett.128.204801 (2022).
doi: 10.1103/PhysRevLett.128.204801
pubmed: 35657864
Roussel, R. et al. Turn-key constrained parameter space exploration for particle accelerators using bayesian active learning. Nat. Commun. 12, 5612 (2021).
doi: 10.1038/s41467-021-25757-3
pubmed: 34556642
pmcid: 8460822
Xu, C. et al. Bayesian optimization for SASE tuning at the European XFEL. In Proc. IPAC’23, no. 14 in IPAC’23 - 14th International Particle Accelerator Conference, 4483–4486, https://doi.org/10.18429/JACoW-IPAC2023-THPL028 (JACoW Publishing, Geneva, Switzerland, 2023).
Kirschner, J. et al. Tuning particle accelerators with safety constraints using bayesian optimization. Phys. Rev. Accel. Beams 25, 062802. https://doi.org/10.1103/PhysRevAccelBeams.25.062802 (2022).
doi: 10.1103/PhysRevAccelBeams.25.062802
Xu, C. et al. Beam trajectory control with lattice-agnostic reinforcement learning. In Proc. IPAC’23, https://doi.org/10.18429/JACoW-IPAC-2023-THPL029 (2023).
Hirlaender, S. et al. Ultra fast reinforcement learning demonstrated at cern awake. In Proc. IPAC’23, no. 14 in IPAC’23 - 14th International Particle Accelerator Conference, 4510–4513, https://doi.org/10.18429/JACoW-IPAC2023-THPL038 (JACoW Publishing, Geneva, Switzerland, 2023).
Krause, A. & Ong, C. Contextual Gaussian process bandit optimization. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F. & Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 24 (Curran Associates, Inc., 2011).
Nyikosa, F. M., Osborne, M. A. & Roberts, S. J. Bayesian optimization for dynamic problems (2018). Preprint available at arXiv:1803.03432 .
Kuklev, N., Sun, Y., Shang, H., Borland, M. & Fystro, G. I. Robust adaptive bayesian optimization. In Proc. IPAC’23, no. 14 in IPAC’23 - 14th International Particle Accelerator Conference, 4377–4380, https://doi.org/10.18429/jacow-ipac2023-thpl007 (JACoW Publishing, Geneva, Switzerland, 2023).
Stein, O., Kaiser, J. & Eichler, A. Accelerating linear beam dynamics simulations for machine learning applications. In Proceedings of the 13th International Particle Accelerator Conference (2022).
Kaiser, J. & Xu, C. Cheetah (2023).
Brockman, G. et al. OpenAI Gym (2016).
Towers, M. et al. Gymnasium https://doi.org/10.5281/zenodo.8127026 (2023).
doi: 10.5281/zenodo.8127026
Fujimoto, S., van Hoof, H. & Meger, D. Addressing function approximation error in actor-critic methods (2018). Preprint available at arXiv:1802.09477v3 .
Raffin, A. et al. Stable Baselines3 (2019).
Balandat, M. et al. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33 (2020).
Matérn, B. Spatial Variation, vol. 36 (Springer New York, 1986), 2 edn.
Jones, D. R., Schonlau, M. & Welch, W. J. Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492. https://doi.org/10.1023/A:1008306431147 (1998).
doi: 10.1023/A:1008306431147
Roussel, R., Edelen, A., Bartnik, A. & Mayes, C. Xopt: A simplified framework for optimization of accelerator problems using advanced algorithms. In Proc. IPAC’23, no. 14 in IPAC’23 - 14th International Particle Accelerator Conference, 4796–4799, https://doi.org/10.18429/jacow-ipac2023-thpl164 (JACoW Publishing, Geneva, Switzerland, 2023).
Fundamental Algorithms for Scientific Computing in Python. Virtanen, P. et al. SciPy 1.0. Nat. Methods17, 261–272 (2020).
Scheinker, A., Huang, E.-C. & Taylor, C. Extremum seeking-based control system for particle accelerator beam loss minimization. IEEE Trans. Control Syst. Technol. 30, 2261–2268. https://doi.org/10.1109/TCST.2021.3136133 (2022).
doi: 10.1109/TCST.2021.3136133