  We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem in
terms of optimizing a real-valued matrix function over the set of feedback
gains. The results developed are in parallel to those in Bu et al. [1] for
discrete-time LTI systems. In this direction, we characterize several
analytical properties (smoothness, coerciveness, quadratic growth) that are
crucial in the analysis of gradient-based algorithms. We also point out
similarities and distinctive features of the continuous time setup in
comparison with its discrete time analogue. First, we examine three types of
well-posed flows direct policy update for LQR: gradient flow, natural gradient
flow and the quasi-Newton flow. The coercive property of the corresponding cost
function suggests that these flows admit unique solutions while the gradient
dominated property indicates that the underling Lyapunov functionals decay at
an exponential rate; quadratic growth on the other hand guarantees that the
trajectories of these flows are exponentially stable in the sense of Lyapunov.
We then discuss the forward Euler discretization of these flows, realized as
gradient descent, natural gradient descent and quasi-Newton iteration. We
present stepsize criteria for gradient descent and natural gradient descent,
guaranteeing that both algorithms converge linearly to the global optima. An
optimal stepsize for the quasi-Newton iteration is also proposed, guaranteeing
a $Q$-quadratic convergence rate--and in the meantime--recovering the
Kleinman-Newton iteration. Lastly, we examine LQR state feedback synthesis with
a sparsity pattern. In this case, we develop the necessary formalism and
insights for projected gradient descent, allowing us to guarantee a sublinear
rate of convergence to a first-order stationary point.
