In the last decade, parallel to the rise of data science and machine learning there has also been a vast growth in the interest and contributions from numerical analysis, scientific computing and computational physics to high-dimensional Bayesian inference, in order to efficiently combine physical models (e.g. PDEs) and data. The aim is a better understanding and control of scientific or engineering problems with a quantitative measure of the remaining uncertainty in actual applications. But what are the problems and challenges for existing statistical approaches that need to be addressed, what are potential alternatives and how can it benefit the field most effectively? In this talk, I will showcase some areas where there are opportunities for numerical analysis and computational physics to have an impact. More specifically, I will present some promising approaches that take inspiration from decades of development in quantum physics to design surrogates that can significantly accelerate Bayesian computation in high-dimensional PDE-constrained applications: multilevel delayed acceptance MCMC [Lykkegaard et al, 2023] and Multigrid Monte Carlo [Kazashi et al, 2023+], as well as a measure-transport approach based on low-rank tensor approximations [Cui et al, 2022].