Tuning particle accelerators is a challenging and time-consuming task, but can be automated and carried out efficiently through the use of suitable optimization algorithms. With successful applications at various facilities, Bayesian optimization using Gaussian process modeling has proven to be a particularly powerful tool to address these challenges in practice. One of its major benefits is that it allows incorporating prior information, such as knowledge about the shape of the objective function or predictions based on archived data, simulations or surrogate models, into the model. In this work, we propose the use of a neural network model as an efficient way to include prior knowledge about the objective function into the Bayesian optimization process to speed up convergence. We report results obtained in simulations and experiments using neural network priors to perform optimization of electron and heavy-ion accelerator facilities, specifically the Linac Coherent Light Source and the Argonne Tandem Linear Accelerator System. Finally, we evaluate how the accuracy of the prior mean predictions affect optimization performance.