  The interpretation of numerical methods, such as finite difference methods
for differential equations, as point estimators suggests that formal
uncertainty quantification can also be performed in this context. Competing
statistical paradigms can be considered and Bayesian probabilistic numerical
methods (PNMs) are obtained when Bayesian statistical principles are deployed.
Bayesian PNM have the appealing property of being closed under composition,
such that uncertainty due to different sources of discretisation in a numerical
method can be jointly modelled and rigorously propagated. Despite recent
attention, no exact Bayesian PNM for the numerical solution of ordinary
differential equations (ODEs) has been proposed. This raises the fundamental
question of whether exact Bayesian methods for (in general nonlinear) ODEs even
exist. The purpose of this paper is to provide a positive answer for a limited
class of ODE. To this end, we work at a foundational level, where a novel
Bayesian PNM is proposed as a proof-of-concept. Our proposal is a synthesis of
classical Lie group methods, to exploit underlying symmetries in the gradient
field, and non-parametric regression in a transformed solution space for the
ODE. The procedure is presented in detail for first and second order ODEs and
relies on a certain strong technical condition -- existence of a solvable Lie
algebra -- being satisfied. Numerical illustrations are provided.
