text: This paper develops a unified methodology for probabilistic analysis and optimal control design for jump diffusion processes defined by polynomials. For such systems, the evolution of the moments of the state can be described via a system of linear ordinary differential equations. Typically, however, the moments are not closed and an infinite system of equations is required to compute statistical moments exactly. Existing methods for stochastic analysis, known as closure methods, focus on approximating this infinite system of equations with a finite dimensional system. This work develops an alternative approach in which the higher order terms, which are approximated in closure methods, are viewed as inputs to a finite-dimensional linear control system. Under this interpretation, upper and lower bounds of statistical moments can be computed via convex linear optimal control problems with semidefinite constraints. For analysis of steady-state distributions, this optimal control problem reduces to a static semidefinite program. These same optimization problems extend automatically to stochastic optimal control problems. For minimization problems, the methodology leads to guaranteed lower bounds on the true optimal value. Furthermore, we show how an approximate optimal control strategy can be constructed from the solution of the semidefinite program. The results are illustrated using numerous examples.
  @cite_13: We consider the information relaxation approach for calculating performance bounds for stochastic dynamic programs DPs. This approach generates performance bounds by solving problems with relaxed nonanticipativity constraints and a penalty that punishes violations of these nonanticipativity constraints. In this paper, we study DPs that have a convex structure and consider gradient penalties that are based on first-order linear approximations of approximate value functions. When used with perfect information relaxations, these penalties lead to subproblems that are deterministic convex optimization problems. We show that these gradient penalties can, in theory, provide tight bounds for convex DPs and can be used to improve on bounds provided by other relaxations, such as Lagrangian relaxation bounds. Finally, we apply these results in two example applications: first, a network revenue management problem that describes an airline trying to manage seat capacity on its flights; and second, an inventory management problem with lead times and lost sales. These are challenging problems of significant practical interest. In both examples, we compute performance bounds using information relaxations with gradient penalties and find that some relatively easy-to-compute heuristic policies are nearly optimal.
  @cite_35: Abstract By using a very simple remark on the moment equations of stochastic Markov processes, one can use the Euler-Lagrange variational approach (with Hamiltonian) to solve some stochastic optimal control problems. As an example one considers a linear-quadratic-Gaussian problem with small state dependent noise, which is solved by a perturbation technique (expansions with respect the small parameter). Another application example is provided by the tracking control of mechanical systems with noisy sliding equations which works via the stochastic Hamilton's principle of mechanical systems. Finally, an economic example shows that the approach also works with moments of fractional order.
  @cite_28: 
  @cite_36: Abstract In order to obtain an approximate solution to the optimal control of nonlinear stochastic systems, one used to suppose that the magnitude of the noise is small enough in such a manner that one can apply a linearization around the deterministic trajectory defined by the system in the absence of noise. When this assumption is not satisfied (i.e. when the magnitude of the noise is of some importance), it is necessary to improve this approximation by taking into account the nonlinear terms of the Taylor's expansion, so that, as a result, we are so dealing with stochastic systems subject to powers of Gaussian white noise. In an engineering mathematics framework, in order to cope with the mathematical difficulties so involved, we propose an approach via the central limit theorem. We first define the integral of powers of Gaussian white noise, whereby we can derive a generalization of the Fokker-Planck-Kolmogorov equation. Then we show how this result can be used in a variational approach to neighbouring stochastic optimal control, via the moment equations.
  @cite_9: We consider the class of nonlinear optimal control problems (OCPs) with polynomial data, i.e., the differential equation, state and control constraints, and cost are all described by polynomials, and more generally for OCPs with smooth data. In addition, state constraints as well as state and or action constraints are allowed. We provide a simple hierarchy of LMI- (linear matrix inequality)-relaxations whose optimal values form a nondecreasing sequence of lower bounds on the optimal value. Under some convexity assumptions, the sequence converges to the optimal value of the OCP. Preliminary results show that good approximations are obtained with few moments.
  @cite_32: 
  @cite_15: 
  @cite_16: 
  @cite_20: We consider the problem of finding the unconstrained global minimum of a real-valued polynomial p(x): R ^n R $, as well as the global minimum of p(x), in a compact set K defined by polynomial inequalities. It is shown that this problem reduces to solving an (often finite) sequence of convex linear matrix inequality (LMI) problems. A notion of Karush--Kuhn--Tucker polynomials is introduced in a global optimality condition. Some illustrative examples are provided.
  @cite_12: 
  @cite_17: This paper approaches optimal control problems for discrete-time controlled Markov processes by representing the value of the problem in a dual Lagrangian form; the value is expressed as an infimum over a family of Lagrangian martingales of an expectation of a pathwise supremum of the objective adjusted by the Lagrangian martingale term. This representation opens up the possibility of numerical methods based on Monte Carlo simulation, which may be advantageous in high-dimensional problems or in problems with complicated constraints.
summary: Most methods for stochastic optimal control focus on dynamic programming @cite_28 @cite_12 or approximate dynamic programming @cite_15 . More closely related to this work, however, are relaxation methods. Unlike approximate dynamic programming, these methods provide bounds on the achievable optimal values. The most closely related methods focus on occupation measures @cite_9 @cite_32 @cite_16 . These works rely on the general insight that a convex relaxation of an optimization problem can be obtained by examining randomized strategies @cite_20 . Such relaxation methods apply to stochastic optimal control problems, but it is not clear that they can be used for analysis problems. In contrast, our relaxation methodology applies to both cases. A more removed relaxation method is studied in @cite_17 @cite_13 . These works use duality to relax the constraint that controllers depend causally on state information. Also related is the work of @cite_35 @cite_36 which uses a combination of moment dynamics and deterministic optimal control to find approximate solutions to stochastic optimal control problems. This work, however, only considers systems with closed moments. Furthermore, this work assumes a fixed parametric form for the controller, and does not obtain a relaxation of the original stochastic control problem.