text:
# Risk-Sensitive Optimal Control of Quantum Systems

## I Introduction

Optimal control theory provides a systematic approach to control system design that is widely used. A cost function is formulated by the designer that encodes the desired performance of the system as its minimum, and then the cost is minimized to obtain the desired controller. Perhaps the most famous example is Kalman’s linear quadratic Gaussian (LQG) regulator problem, where the cost criterion is an average of an integral,

$$J^{LQG}=𝐄[\underset{k=0}{\overset{M1}{}}(x_k^{}Px_k+u_k^{}Qu_k+x_M^{}P_Mx_M)]$$
(1)
where $`x_k`$ and $`u_k`$ are respectively state and control variables (vectors), and $`P`$, $`Q`$ and $`P_M`$ are weighting matrices 111If $`x`$ is a column vector, the notation $`x^{}`$ indicates the transpose, a row vector.. The cost criterion (1) is an example of what is sometimes called a risk-neutral criterion. The state (or phase space) variable $`x_k`$ is part of the model of the classical physical system being controlled. In general, the controller has only partial access to state information, with measurements corrupted by noise. Kalman’s optimal LQG feedback controller is an explicit function of the conditional state and covariance. It is dynamic, since the conditional state and covariance evolve in time via the Kalman Filter (see, e.g., KA70 , KV86 ), and the Kalman filter does not involve the cost function in any way; it gives the optimal mean square state estimate independently of any control objective. Interestingly, the function giving the optimal feedback control is the same as for an analogous problem with full state information, viz. multiplication by a gain matrix determined by solving a Riccati equation. Kalman’s optimal LQG controller is the paradigm example of the so-called separation structure, where the controller is decomposed into an estimation part (filtering) and a control part, as illustrated in Figure 1.

Over the past 20 or so years, another type of optimal control problem has generated considerable interest, viz. linear exponential quadratic Gaussian (LEQG) optimal control, or risk-sensitive optimal control, J73 ; W81 ; BV85 . In this average of exponential of integral problem, the cost is of the form

$$\begin{array}{c}J^{LEQG}=\hfill \\ 𝐄[\mathrm{exp}\left(_{k=0}^{M1}(x_k^{}Px_k+u_k^{}Qu_k)+x_M^{}P_Mx_M\right)].\hfill \end{array}$$
(2)
In this case the optimal feedback control is an explicit function of a dynamical quantity closely related to the conditional state and covariance, but given by dynamics that include terms from the cost function. It also has a separation structure, though in this case the filter depends on the cost function used to specify the performance objective, and is a modification of the Kalman Filter. One of the major reasons for the interest in the risk-sensitive problem is its close connections to robust control and minimax games, GD88 ; DGKF89 ; JBE94 . Robust control concerns the desire to design controllers that are robust with respect to uncertainty, such as model errors and exogenous disturbances, GL95 . Robustness properties of risk-sensitive controllers are described in DJP00 .

Risk-neutral, risk-sensitive and other stochastic control problems have been considered for problems with a finite number of states, see, e.g. KV86 ; BJ97 ; FGM94 ; CM95 . After an analysis of an example of a machine replacement problem CM95 , the authors concluded that for that problem the risk-neutral controller was more aggressive than risk-sensitive and related minimax controllers.

Suppose we wish to control a quantum physical system using real time feedback via a non-quantum feedback system (say using a digital computer) in some optimal fashion. If one were to do this using a standard cost criterion, say one analogous to Kalman’s (LQG) regulator problem (risk-neutral), then one would find that the optimal control is a function of the conditional (selective) state (a density operator), as is well known, see, e.g. VPB83 ; DJ99 ; DHJMT00 ; VDM01 . The conditional state is the solution of a stochastic master equation that describes the evolution of our knowledge of the system. This stochastic master equation is used in two ways: (i) as the model of the quantum physical system, taking into account the effect of the measurements, and (ii) as the dynamics of the filter in the optimal controller, Figure 1.

The purpose of this paper is to consider the risk-sensitive optimal control of quantum physical systems. The quantum systems are modelled by stochastic master equations for the conditional state. The risk-sensitive criterion is one of a class of multiplicative cost functions. The optimal solution for this class of problems has a separation structure, Figure 1, where the filter describes the evolution of an unnormalized conditional state via a modified stochastic master equation that contains the cost function used to specify the performance objective. The optimal control is a function of this unnormalized conditional state. It is important to note that, in contrast to the risk-neutral case described above, the states and dynamics for the quantum physical model and the filter are not the same. Indeed, the unnormalized conditional dynamic equation used in the filter describes the evolution of our knowledge of the quantum system tempered by our purpose for the controlled quantum system. This type of extension of the conditional dynamics appears to be new to quantum physics, and may merit further investigation. We emphasize that the unnormalized conditional state is defined only in the context of the risk-sensitive and multiplicative control objectives considered here, where it is used in a specific feedback situation. Again, we emphasize that (i) the model of the quantum physical system is the standard stochastic master equation for the conditional state, and (ii) the filter is described by a modified stochastic master equation for an unnormalized conditional state; this modified equation includes terms from the cost function.

This paper is organized as follows. In section II we carefully describe the model we use for the controlled quantum system. Then in section III we summarize some relevant results for a risk-neutral optimal control problem, and make some comments on the feedback solution. Section IV contains the formulation and dynamic programming solution to the risk-sensitive and related multiplicative cost optimal control problems, together with a brief discussion of robustness. The ideas are illustrated by a simple example of a two-state system with feedback. Further developments, applications and examples will be given in subsequent papers.

Acknowledgement. The author wishes to thank Professor I.R. Petersen for many valuable discussions.

## II The Controlled Quantum System

### II.1 Controlled State Transfer

We consider a controlled quantum physical system with inputs $`u`$ and outputs $`y`$. The inputs represent signals or actions that are applied to the system, such as voltages, forces, or light pulses. The outputs are signals that result from repeated measurements of observable quantities, such as position, spin, etc. We will assume, for simplicity, that the measurements are discrete valued. It is sometimes useful to denote the range of input and output values by $`𝐔`$ and $`𝐘`$ respectively.

The state of the quantum system is described by a density operator $`\omega `$222Without further qualification, we use the term state to refer to a positive self-adjoint operator normalized to have trace equal to one.. This state evolves in time as a result of a variety of factors including the underlying unitary evolution, interaction with the environment, the effect of repeated measurements, and feedback control actions. Since measurements are made, and the outcomes are used to determine control actions in a feedback context, we are interested in the selective or conditional evolution of the states. As an example BB91 ; VPB01 ; WM94 , a range of conditional evolutions can be described by an Ito-type stochastic master equations (SME) of the form

$$d\omega =[\omega ]dt+[\omega ]dW$$
(3)
for suitable (super) operators $``$ and $``$ (which may depend on the control $`u`$). Here, $`dW`$ represents an Ito-type Brownian motion (Wiener process) increment, called an innovation, related to the measured output value $`y`$ by

$$dy=\mathrm{tr}\{𝒩[\omega ]\}dt+dW$$
(4)
for a suitable (super) operator $`𝒩`$. If we denote by $`\rho `$ the expected value of $`\omega `$ with respect to $`W`$ (or $`y`$), we obtain the master equation, frequently encountered in the analysis of open systems:

$$\dot{\rho }=[\rho ].$$
(5)
It is conceptually and technically simpler to work in discrete time, and so we will do so in this paper. Effectively, we will be using a model for sampled-data feedback control of quantum systems. In this model, measurements are made and control actions are applied at discrete time instants $`t_k`$333For example, $`t_k=k\mathrm{\Delta }t`$, where $`\mathrm{\Delta }t`$ is the time between samples, and $`k`$ is an integer indicating discrete time values., called sample times. Continuous time models are of considerable importance, and will be considered elsewhere.

The discrete time model we use for the quantum system is defined in terms of a (super) operator $`\mathrm{\Gamma }(u,y)`$ 444The operator $`\mathrm{\Gamma }(u,y)`$ should preserve self-adjointness and positivity. that depends on the control input $`u`$ and the output measurement $`y`$. The idea is that if the quantum system is in state $`\omega _k`$ at time $`k`$, and at this time the control value $`u_k`$ is applied, a measurement outcome $`y_{k+1}`$ will be recorded, and the system will transfer to a new state $`\omega _{k+1}`$. The probability of $`y_{k+1}`$ is $`p(y_{k+1}|u_k,\omega _k)`$, where

$$p(y|u,\omega )=\mathrm{\Gamma }(u,y)\omega ,I.$$
(6)
Here, we have used the notation

$$\omega ,B=\mathrm{tr}[B\omega ]$$
(7)
to specify the (expected) value of an observable $`B`$ when the system is in state $`\omega `$. The operator $`\mathrm{\Gamma }(u,y)`$ is assumed to be normalized, i.e.

$$\underset{y𝐘}{}\mathrm{\Gamma }(u,y)\omega ,I=\omega ,I=1$$

so that $`p(y|u,\omega )`$ is a probability distribution, since it satisfies $`_yp(y|u,\omega )=1`$.

Selective or conditional evolution means that the new state $`\omega _{k+1}`$ depends on the value of the measurement $`y_{k+1}`$, and we write this dependance as follows:

$$\omega _{k+1}=\mathrm{\Lambda }_\mathrm{\Gamma }(u_k,y_{k+1})\omega _k,$$
(8)
where

$$\mathrm{\Lambda }_\mathrm{\Gamma }(u,y)\omega =\frac{\mathrm{\Gamma }(u,y)\omega }{p(y|u,\omega )}.$$
(9)
Equation (8) is a discrete time stochastic master equation (SME), and can be viewed, e.g., as the result of integrating an equation of the form (3) over one time step (after substituting for $`dW`$ in terms of $`dy`$).

We denote the average of the conditional state $`\omega _k`$ with respect to the measurements by $`\rho _k`$. If $`u_k`$ is a deterministic (non-random) input signal, then $`\rho _k`$ satisfies the master equation

$$\rho _{k+1}=\underset{y𝐘}{}\mathrm{\Gamma }(u_k,y)\rho _k.$$
(10)
Equation (8) constitutes our model of the quantum system. For further information on this framework of operator valued measures and quantum operations, see EBD76 ; NC00 ; GZ00 ; VPB83 . We now give some examples.

###### Example II.1

We define the controlled transfer $`\mathrm{\Gamma }(u,y)`$ by interleaving open system dynamics and imperfect orthogonal measurements. The open system dynamics are modelled by a quantum operation

$$^u\omega =\underset{b}{}E_b^u\omega E_b^u$$
(11)
where the controlled operators $`E_b^u`$ satisfy $`_bE_b^uE_b^u=I`$ for all inputs $`u`$. Closed systems are described by the unitary evolution operation $`^u\omega =T^u\omega T^u`$, where for each input value $`u`$, $`T^u`$ is a unitary operator.

The imperfect measurements are modelled as follows. Let $`A`$ be a self-adjoint operator with discrete nondegenerate spectrum $`\mathrm{spec}(A)`$. For $`a\mathrm{spec}(A)`$ an eigenvalue of $`A`$ let $`|a`$ denote the normalized eigenvector, and let $`P_a=|aa|`$ denote the projection onto the eigenspace of $`A`$ ($`P_a|\psi =a|\psi |a`$). Perfect measurements would correspond to $`y=a`$; however, to reflect the presence of measurement noise in applications we will assume that when a measurement occurs on the quantum system, the values $`a`$ and associated projections occur in the usual (perfect) way, but that knowledge of the outcomes is corrupted by sensor noise so that the controller (or any observing device or person) measures a value $`y`$. The measurement $`y`$ is a random variable, related to the outcomes $`a`$ via probability kernels $`q(y|a)`$, the probability of $`y`$ given that $`a`$ occurred. The kernels have the property that $`_yq(y|a)=1`$ for all $`a`$. In the case of perfect measurements, $`q(y|a)=1`$ if $`y=a`$, and $`q(y|a)=0`$ if $`ya`$.

The operator $`\mathrm{\Gamma }(u,y)`$ is given by

$$\mathrm{\Gamma }(u,y)\omega =\underset{a,b}{}q(y|a)P_aE_b^u\omega E_b^uP_a$$
(12)
and the adjoint is given by

$$\mathrm{\Gamma }^{}(u,y)B=\underset{a,b}{}q(y|a)E_b^uP_aBP_aE_b^u$$
(13)
where $`B`$ is an observable. The expressions in this example can be derived using standard techniques of quantum operations and discrete time filtering based on Bayes’ Rule (see, e.g., (GZ00, , Chapter 2.2), (NC00, , Chapter 8), EBD76 , VPB83 , (KV86, , Chapter 6), (KA70, , Chapter 7)). $`\mathrm{}`$

###### Example II.2

(Two-state system.) We now describe a specific instance of Example II.1, viz. a two-state system and measurement device, where it is desired to use feedback control to put the system into a given state. The example is inspired by a simple quantum feedback example (HW94, , Section 1.3) and an example in stochastic control concerning a machine replacement problem FGM94 ; CM95 .

In (HW94, , Section 1.3), a particle beam is passed through a Stern-Gerlach device, which results in one beam of particles in the up state, and one beam in the down state. The beam of particles in the up state is subsequently left alone, while the beam in the down state is subject to a further device which will result in a change of spin direction from down to up. The final outcome of this feedback arrangement is that all particles are in the up state. Analogous feedback configurations can be constructed using other physical systems, e.g. light and polarization measurement.

In what follows we extend the general features of this example to accommodate repeated noisy measurements. Physically, the noisy measurements might arise from imperfectly separated beams, where a proportion of each beam contaminates the other, and/or from interference or noise affecting sensors. The example was chosen because the risk-neutral and risk-sensitive problems can be solved explicitly. Hence the example provides a concrete illustration of some ideas concerning quantum feedback control. More substantial examples and applications will be considered elsewhere.

The pure states of the system are of the form

$$|\psi =c_1|1+c_1|1\left(\begin{array}{c}c_1\\ c_1\end{array}\right).$$

The states $`|1`$ and $`|1`$ are eigenstates of the observable

$$A=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$$
(14)
corresponding to ideal measurement values $`a=1`$ and $`a=1`$. It is desired to put the system into the state

$$|1=\left(\begin{array}{c}0\\ 1\end{array}\right),\text{or}|11|=\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right).$$

We define a controlled transfer operator $`\mathrm{\Gamma }(u,y)`$ as the following physical process, Figure 2. First apply a unitary transformation $`T^u`$, where the control value $`u=0`$ means do nothing, while $`u=1`$ means to flip the states (quantum not gate), i.e.

$$T^u=\{\begin{array}{cc}\hfill \left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)& \text{if}u=0\hfill \\ \hfill \left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)& \text{if}u=1.\hfill \end{array}$$

We then make an imperfect measurement corresponding to the observable $`A`$. We model this by an ideal device (e.g. Stern-Gerlach, beam splitter) with projection operators

$$P_1=\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right),P_1=\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right)$$

followed by a memoryless channel with error probability kernels

$$\begin{array}{cc}q(1|1)\hfill & =1\alpha \hfill \\ q(1|1)\hfill & =\alpha \hfill \\ q(1|1)\hfill & =\alpha \hfill \\ q(1|1)\hfill & =1\alpha \hfill \end{array}$$

where $`0\alpha 1`$ is the probability of a measurement error (cf. (NC00, , Figure 8.1)).

The controlled transfer operator is therefore (from (12))

$$\begin{array}{c}\mathrm{\Gamma }(u,y)\omega =q(y|1)P_1T^u\omega T^uP_1\hfill \\ +q(y|1)P_1T^u\omega T^uP_1.\hfill \end{array}$$

In this example, the control $`u`$ can take the values $`0`$ or $`1`$, and output $`y`$ has values $`0`$ or $`1`$ ($`𝐔=\{0,1\})`$, $`𝐘=\{0,1\})`$.

If we write a general density matrix as

$$\omega =\left(\begin{array}{cc}\omega _{11}& \omega _{12}\\ \omega _{12}^{}& \omega _{22}\end{array}\right),$$
(15)
then the controlled operators $`\mathrm{\Gamma }(u,y)`$ are given explicitly by

$$\begin{array}{cc}\hfill \mathrm{\Gamma }(0,1)\omega & =\left(\begin{array}{cc}(1\alpha )\omega _{11}& 0\\ 0& \alpha \omega _{22}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }(0,1)\omega & =\left(\begin{array}{cc}\alpha \omega _{11}& 0\\ 0& (1\alpha )\omega _{22}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }(1,1)\omega & =\left(\begin{array}{cc}(1\alpha )\omega _{22}& 0\\ 0& \alpha \omega _{11}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }(1,1)\omega & =\left(\begin{array}{cc}\alpha \omega _{22}& 0\\ 0& (1\alpha )\omega _{11}\end{array}\right)\hfill \end{array}$$

This example is continued in stages in the remainder of the paper (Examples II.3, III.1, III.3, IV.7). $`\mathrm{}`$

### II.2 Feedback Control

In the above description of the quantum system (8), we have not described how the controls $`u_k`$ are determined by the measurements $`y_k`$ via a feedback controller $`K`$. We now do this.

Feedback controllers should be causal, i.e., the current control value $`u_k`$ cannot depend on future values of the measurements $`y_{k+1},y_{k+2},\mathrm{}`$. On a time interval $`0kM1`$ this is expressed as follows:

$$K=\{K_0,K_1,\mathrm{},K_{M1}\}$$

where

$$\begin{array}{cc}\hfill u_0& =K_0\hfill \\ \hfill u_1& =K_1(y_1)\hfill \\ \hfill u_2& =K_2(y_1,y_2)\hfill \\ \hfill \text{etc.}& \end{array}$$

To simplify notation, we often write sequences $`u_{k_1},u_{k_1+1},\mathrm{},u_{k_2}`$ as $`u_{k_1,k_2}`$. Then we can write $`u_k=K_k(y_{1,k})`$. A controller $`K`$ can be restricted to subintervals $`kjM`$ by fixing (or omitting) the first arguments in the obvious way. We denote by $`𝒦`$ the class of all such feedback controllers.

A feedback controller $`K`$ in closed loop with the quantum system, Figure 3, operates as follows. The given initial state $`\omega _0`$ and controller $`K`$ are sufficient to define random sequences of states $`\omega _{0,M}`$, inputs $`u_{0,M1}`$ and outputs $`y_{1,M}`$ over a given time interval $`0kM`$ iteratively as follows. The control value $`u_0`$ is determined by $`K_0`$ (no observations are involved yet), and it is applied to the quantum system, which responds by selecting $`y_1`$ at random according to the distribution $`p(y_1|u_0,\omega _0)`$. This then determines the next state $`\omega _1`$ via (8). Next $`u_1`$ is given by $`K_1(y_1)`$, and applied to the system. This process is repeated until the final time.

The controller $`K`$ therefore determines controlled stochastic processes $`\omega _k`$, $`u_k`$ and $`y_k`$ on the interval $`0kM`$. Expectation with respect to the associated probability distribution is denoted $`𝐄_{\omega _0,0}^K`$. The state sequence $`\omega _k`$ is a controlled Markov process.

One way a controller $`K`$ can be constructed is using a function

$$u_k=𝐮(\omega _k,k)$$

where $`\omega _k`$ is given by (8) with initial state $`\omega _0`$. This controller is denoted $`K_{\omega _0}^𝐮`$. The SME equation (8) forms part of this controller, viz. its dynamics, and must be implemented with suitable technology (e.g. digital computer). Controllers of this type are said to have a separation structure, where the controller can be decomposed into an estimation part (i.e. filtering via (8)) and a control part (i.e. the function $`𝐮`$). We will see in section III that the optimal risk-neutral controller is of this form (Figure 6). In section IV, the optimal risk-sensitive controller also has a separation structure, but the filter used is different (Figure 7). The separation structure arises naturally from the dynamic programming techniques, as we shall see.

###### Example II.3

(Two-state system with feedback, Example II.2 continued.) We consider a particular feedback controller $`\overline{K}`$ for a time horizon $`M=2`$ defined by

$$u_0=\overline{K}_0=0,u_1=\overline{K}_1(y_1)=\{\begin{array}{cc}\hfill 0& \text{if}y_1=1\hfill \\ \hfill 1& \text{if}y_1=1.\hfill \end{array}$$
(16)
We apply $`\overline{K}`$ to the system with initial pure state

$$|\psi _0=\frac{1}{\sqrt{2}}|1+\frac{1}{\sqrt{2}}|1,\text{or}\omega _0=\frac{1}{2}\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right).$$
(17)
The result is shown in Table 1, which displays the resulting conditional states

$$\begin{array}{cc}\hfill \omega _1^{(u_0,y_1)}& =\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\omega _0,\hfill \\ \hfill \omega _2^{(u_0,y_1),(u_1,y_2)}& =\mathrm{\Lambda }_\mathrm{\Gamma }(u_1,y_2)\omega _1^{(u_0,y_1)}\hfill \end{array}$$

and the associated probabilities. Explicitly, the terms shown in Table 1 are:

$$p_1=p(y_1|u_0,\omega _0),p_2=p(y_2|u_1,\omega _1)$$

$$\omega _1^{(0,1)}=\left(\begin{array}{cc}(1\alpha )& 0\\ 0& \alpha \end{array}\right),\omega _1^{(0,1)}=\left(\begin{array}{cc}\alpha & 0\\ 0& (1\alpha )\end{array}\right)$$

$$\omega _2^{(0,1),(1,1)}=\omega _2^{(0,1),(0,1)}=\frac{1}{2}\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),$$

$$\begin{array}{c}\omega _2^{(0,1),(1,1)}=\omega _2^{(0,1),(0,1)}\hfill \\ =\frac{1}{\alpha ^2+(1\alpha )^2}\left(\begin{array}{cc}\alpha ^2& 0\\ 0& (1\alpha )^2\end{array}\right).\hfill \end{array}$$

Also shown are the non-selective states:

$$\rho _0=\omega _0$$

$$\begin{array}{cc}\hfill \rho _1& =p(1|u_0,\omega _0)\omega _1^{(0,1)}+p(1|u_0,\omega _0)\omega _1^{(0,1)}\hfill \\ & =\frac{1}{2}\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\hfill \end{array}$$

$$\begin{array}{c}\rho _2=p(1|1,\omega _1^{(0,1)})\omega _2^{(0,1),(1,1)}\hfill \\ +p(1|1,\omega _1^{(0,1)})\omega _2^{(0,1),(1,1)}\hfill \\ +p(1|0,\omega _1^{(0,1)})\omega _2^{(0,1),(0,1)}\hfill \\ +p(1|0,\omega _1^{(0,1)})\omega _2^{(0,1),(0,1)}\hfill \\ =\frac{1}{2}\left(\begin{array}{cc}\alpha ^2+\alpha (1\alpha )& 0\\ 0& \alpha (1\alpha )+(1\alpha )^2\end{array}\right).\hfill \end{array}$$
(18)
At time $`k=0`$ the control $`u=0`$ is applied. If $`y_1=1`$ is observed, as a result of the imperfect measurement, the system moves to the state $`\omega _1^{(0,1)}`$. Since $`y_1=1`$, the controller $`\overline{K}`$ (16) gives $`u_1=1`$. This results in the states $`\omega _2^{(0,1),(1,1)}`$ or $`\omega _2^{(0,1),(1,1)}`$, depending on the outcome of the second measurement $`y_2`$. If, on the other hand, $`y_1=1`$ is observed, the system moves to the state $`\omega _1^{(0,1)}`$. Since $`y_1=1`$, the controller $`\overline{K}`$ (16) gives $`u_1=0`$, and hence $`\omega _2^{(0,1),(0,1)}`$ or $`\omega _2^{(0,1),(0,1)}`$, again depending on the outcome of the second measurement $`y_2`$. This is illustrated in Figure 4.

These results are consistent with (HW94, , Section 1.3). Indeed, when $`\alpha =0`$ (perfect measurements), the feedback system terminates in the desired pure state $`\rho _2=|11|`$. The role of feedback control is clearly demonstrated here. With imperfect measurements, $`0<\alpha <1`$, the system terminates in the mixed state $`\rho _2`$ given by (18), with the degree of mixing (indicating the expected degradation in performance) depending on the measurement error probability parameter $`\alpha `$:

$$\begin{array}{cc}\hfill \mathrm{tr}\rho _2^2& =(\alpha ^2+\alpha (1\alpha ))^2+(\alpha (1\alpha )+(1\alpha )^2)^2\hfill \\ & <1\text{if}0<\alpha <1\hfill \\ & =1\text{if}\alpha =0.\hfill \end{array}$$

$`\mathrm{}`$

## III Risk-Neutral Control

In this section we summarize dynamic programming results for a well-known type of finite time horizon optimal control problem, VPB83 ; KV86 . The optimal control problem discussed here can be considered to be a prototype problem illustrating measurement feedback in the quantum context. The dynamic programming methods used in this paper for solving the optimal control problems are standard, and the reader is referred to the literature for further information, see, e.g. KA70 ; KV86 ; DB95 .

We define a cost function to be a non-negative observable $`L(u)`$ that can depend on the control $`u`$. The cost function encodes the designer’s control objective. We also use a non-negative observable $`N`$ to define a cost for the final state.

###### Example III.1

(Two-state system with feedback, Example II.3 continued.) To set up the cost function $`L(u)`$ to reflect our objective of regulating the system to the desired pure state $`|1`$, we define

$$X=\frac{1}{2}(A1.I)=\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right)$$

where $`A`$ is the observable corresponding to the projective measurement (14). We note that the expected value of $`X^2`$ is

$$\begin{array}{ccc}\hfill 1|X^2|1& =\mathrm{tr}[X^2|11|]\hfill & =0\hfill \\ \hfill 1|X^2|1& =\mathrm{tr}[X^2|11|]\hfill & =1\hfill \end{array}$$

which gives zero cost to the desired state, and nonzero cost to the undesired state. We shall also introduce a cost of control action, as follows:

$$c(u)=\{\begin{array}{cc}\hfill 0& \text{if}u=0\hfill \\ \hfill p& \text{if}u=1\hfill \end{array}$$

where $`p>0`$. This gives zero cost for doing nothing, and a nonzero cost for the flip operation. Thus we define the cost function to be

$$L(u)=X^2+c(u)I$$
(19)
and the cost for the final state is defined to be

$$N=X^2.$$

This modifies our earlier objective of putting the system into the desired state by including a penalty for control action. $`\mathrm{}`$

Let $`M>0`$ be a positive integer indicating a finite time interval $`k=0,\mathrm{},M`$. Given a sequence of control values $`u_{0,M1}=u_0,\mathrm{},u_{M1}`$ and measurements $`y_{1,M}=y_1,\mathrm{},y_M`$, define the risk-neutral cost functional

$$J_{\omega ,0}(K)=𝐄_{\omega ,0}^K[\underset{i=0}{\overset{M1}{}}\omega _i,L(u_i)+\omega _M,N],$$
(20)
where $`\omega _i`$, $`i=0,\mathrm{},M`$ is the solution of the system dynamics (8) with initial state $`\omega _0=\omega `$ under the action of a controller $`K`$. This is an appropriate quantum generalization of the classical LQG cost (1). The objective is to minimize this functional over all measurement feedback controllers $`K𝒦`$.

Following VPB83  it is convenient to rewrite the cost functional (20). For each $`k`$, given a sequence of control values $`u_{k,M1}=u_k,\mathrm{},u_{M1}`$ and measurements $`y_{k+1,M}=y_{k+1},\mathrm{},y_M`$, define a random sequence of observables $`Q_k`$ by the recursion ((VPB83, , equation (3.1)))

$$\begin{array}{cc}\hfill Q_k& =\mathrm{\Gamma }^{}(u_k,y_{k+1})Q_{k+1}+L(u_k),0kM1\hfill \\ \hfill Q_M& =N\hfill \end{array}$$
(21)
When useful, we write

$$Q_k=Q_k(u_{k,M1},y_{k+1,M})$$

to indicate dependence on the input and outputs. $`Q_k`$ may be called a cost observable. The cost functional (20) is given by

$$\begin{array}{c}J_{\omega ,0}(K)\hfill \\ =\underset{y_{1,M}𝐘^M}{}\omega ,Q_0(K(y_{1,M})_{0,M1},y_{1,M})\hfill \end{array}$$
(22)
Here and elsewhere we use abbreviations of the form

$$K(y_{1,M})_{0,M1}=(K_0,K_1(y_1),\mathrm{},K_{M1}(y_{1,M1}))$$

###### Remark III.2

The cost observable $`Q_k`$ given by (21) and the expression in (22) is analogous to the familiar Heisenberg picture used in quantum physics. It is very natural from the point of view of dynamic programming, and indeed (20) and (22) are related by iterating (21). Here is the first step:

$$\begin{array}{cc}\hfill \omega _0,Q_0& =\omega _0,\mathrm{\Gamma }^{}(u_0,y_1)Q_1+L(u_0)\hfill \\ & =\omega _0,L(u_0)+\mathrm{\Gamma }(u_0,y_1)\omega _0,Q_1\hfill \\ & =\omega _0,L(u_0)+\omega _1,Q_1p(y_1|u_0,\omega _0)\hfill \end{array}$$

where $`\omega _1=\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\omega _0`$ and $`p(y_1|u_0,\omega _0)`$ is given by (6). $`\mathrm{}`$

The key idea of dynamic programming is to look at the current state at a current time $`0kM1`$ and to optimize the remaining cost from the current time to the final time. This leads to an iterative solution. Accordingly, we define, for each $`0kM`$, the cost to go incurred by a controller $`K`$ (restricted to $`klM1`$) to be

$$\begin{array}{c}J_{\omega ,k}(K)\hfill \\ =\underset{y_{k+1,M}𝐘^{Mk}}{}\omega ,Q_k(K(y_{k+1,M})_{k,M1},y_{k+1,M})\hfill \end{array}$$
(23)
The dynamic programming equation associated with this risk-neutral problem is

$$\begin{array}{cc}\hfill V(\omega ,k)& =\underset{u𝐔}{inf}\{\omega ,L(u)\hfill \\ & +\underset{y𝐘}{}V(\mathrm{\Lambda }_\mathrm{\Gamma }(u,y)\omega ,k+1)p(y|u,\omega )\},\hfill \\ \hfill V(\omega ,M)& =\omega ,N\hfill \end{array}$$
(24)
where $`0kM1`$. This is the fundamental equation from which optimality or otherwise of a controller can be determined.

Let $`V`$ be the solution to the dynamic programming equation (24). Then for any controller $`K𝒦`$ we have

$$V(\omega ,k)J_{\omega ,k}(K).$$
(25)
If we assume in addition that a minimizer

$$\begin{array}{cc}\hfill 𝐮^{}(\omega ,k)& \underset{u𝐔}{argmin}\{\omega ,L(u)\hfill \\ & +\underset{y𝐘}{}V(\mathrm{\Lambda }_\mathrm{\Gamma }(u,y)\omega ,k+1)p(y|u,\omega ))\}\hfill \end{array}$$
(26)
exists555The notation $`argmin_{u𝐔}f(u)`$ means the subset of values from $`𝐔`$ minimizing $`f`$. for all $`\omega `$, $`0kM1`$, then the separation structure controller $`K_{\omega _0}^𝐮^{}`$ (recall section II.2) defined by (26) is optimal, i.e.

$$J_{\omega _0,0}(K_{\omega _0}^𝐮^{})=V(\omega _0,0)J_{\omega _0,0}(K)$$
(27)
for all $`K𝒦`$.

###### Example III.3

(Two-state system with feedback, Example III.1 continued.) We solve the dynamic programming equation (24) and determine the optimal feedback controls as follows. For $`k=M=2`$ we have

$$V(\omega ,2)=\omega ,X^2=\omega _{11}$$

and hence for $`k=1`$

$$V(\omega ,1)=\omega _{11}+\mathrm{min}[V_0(\omega ,1),V_1(\omega ,1)]$$

where where $`V_0(\omega ,1),V_1(\omega ,1)`$ are given in Appendix A. Hence we obtain

$$𝐮^{}(\omega ,1)=\{\begin{array}{cc}\hfill 0& \text{if}V_0(\omega ,1)V_1(\omega ,1)\hfill \\ \hfill 1& \text{if}V_0(\omega ,1)>V_1(\omega ,1).\hfill \end{array}$$

At time $`k=0`$ we have

$$V(\omega ,0)=\omega _{11}+\mathrm{min}[V_0(\omega ,0),V_1(\omega ,0)]$$

where $`V_0(\omega ,0),V_1(\omega ,0)`$ are given in Appendix A, which gives

$$𝐮^{}(\omega ,0)=\{\begin{array}{cc}\hfill 0& \text{if}V_0(\omega ,0)V_1(\omega ,0)\hfill \\ \hfill 1& \text{if}V_0(\omega ,0)>V_1(\omega ,0).\hfill \end{array}$$

The optimal risk-neutral feedback controller is given by

$$u_0=K_{\omega _0,0}^𝐮^{}=𝐮^{}(\omega _0,0),u_1=K_{\omega _0,1}^𝐮^{}(y_1)=𝐮^{}(\omega _1,1)$$

where $`\omega _1=\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\omega _0`$. Note that the control $`u_1`$ depends on $`y_1`$ through the conditional state $`\omega _1`$ (separation structure). A physical implementation of the quantum system with optimal risk-neutral feedback is shown in Figure 5.

Let’s consider the special case $`\alpha =0`$ and $`p=0`$, with initial state (17). We then find that $`V_0(\omega _0,0)=V_1(\omega _0,0)=0.5`$, and hence we take $`𝐮^{}(\omega _0,0)=0`$; i.e. $`u_0=0`$.

Next, if $`y_1=1`$ is observed, we have $`\omega _1=|11|`$, $`V_0(\omega _1,1)=1`$ and $`V_1(\omega _1,1)=0`$. Hence we take $`𝐮^{}(\omega _1,1)=1`$, i.e. $`u_1=1`$. However, if $`y_1=1`$ is observed, we have $`\omega _1=|11|`$, $`V_0(\omega _1,1)=0`$ and $`V_1(\omega _1,1)=1`$; and hence we take $`𝐮^{}(\omega _1,1)=0`$, i.e. $`u_1=0`$. In either case we achieve the desired state $`\rho _2=\omega _2=|11|`$.

This action is the same as that seen before for the controller $`\overline{K}`$. The same controller is obtained for $`0<\alpha <0.5`$ and $`p=0`$, but $`\omega _2`$ will be a mixed state. If $`p0`$ the optimal controller $`K_{\omega _0}^𝐮^{}`$ will result in control actions that in general differ from those of $`\overline{K}`$. $`\mathrm{}`$

###### Remark III.4

Note that the optimal risk-neutral controller $`K_{\omega _0}^𝐮^{}`$ determined by (26) feeds back the conditional state $`\omega _k`$, given by the SME (8), in accordance with its separation structure, Figure 6. We note that the conditional state $`\omega _k`$ obtained from (8) provides the optimal means for calculating estimates of observables (in the sense of minimum mean square error), viz. $`\omega _k,B`$, and can be regarded as the optimal filter in this sense. This means that from the point of view of optimal risk-neutral control, the best thing to do is to make use of the optimal filter (8), a dynamical quantity that contains knowledge of the quantum system, as obtained by the controller through the measurement process embedded in $`\mathrm{\Gamma }`$. $`\mathrm{}`$

###### Remark III.5

A second remark we wish to make here concerns the well-known concept in control engineering of dual control, (KV86, , Chapter 6.8). This concept relates to the dual function of the measurement feedback controller $`K`$, viz. (i) to alter the future evolution of the system, and (ii) to alter the future values of the available information. The optimal choice of $`K`$ takes both of these factors into account. $`\mathrm{}`$

###### Remark III.6

The final remark for this section concerns feedback and robustness. Feedback is the most important concept in control engineering, and has a long history going back at least to the mechanical governors used to regulate the speed of steam engines. Feedback is used to compensate for disturbances and uncertainty, and feedback loops typically enjoy a robustness margin (e.g. gain margin and phase margin in classical control engineering), a measure of this compensation ability. Note that in the absence of disturbances and uncertainty, feedback is completely unnecessary and control can be achieved by a prescribed open loop controller. However, in reality both quantum and classical systems are subject to disturbances and uncertainty, e.g. (i) the influence of an environment, (ii) model error due to approximation and unknown parameters, and (iii) imprecise measurements. In the quantum context, there is the further complication DJJ00  that as a consequence of the act of measurement randomness is introduced, and this could potentially reduce control effectiveness. Measurement feedback control of quantum systems is fundamentally a stochastic control problem containing non-classical characteristics. These considerations underscore the importance of feedback control and the need for robustness when controlling quantum systems. Robustness issues will be taken up again in section IV (see Example IV.8). $`\mathrm{}`$

## IV Multiplicative Costs and Risk-Sensitive Control

We turn now to the risk-sensitive optimal control problem, the main object of this paper. The risk-sensitive cost functional we consider, a quantum generalization of LEQG (2), is

$$J_{\omega ,0}^\mu (K)=𝐄_{\omega ,0}[\underset{k=0}{\overset{M1}{}}\omega _k,e^{\mu L(u_k)}\omega _M,e^{\mu N}]$$
(28)
where $`\mu >0`$ is a positive risk parameter, $`L(u)`$ is a cost function (as defined in section III), and $`N`$ is a non-negative observable. The conditional states $`\omega _k`$ are given by the quantum system model (8).

###### Remark IV.1

The risk-sensitive cost (28), by use of the exponential function, gives heavy weight to large values of the cost functions in the exponents. A system controlled by a controller minimizing this cost is not likely to experience large values of these quantities. Risk-sensitive controllers are known to enjoy some robustness properties against uncertainty in the model and external disturbances, see DJP00  and Example IV.8. $`\mathrm{}`$

The primary goal in this section is to find the optimal controller for the risk-sensitive cost functional (28). As noted, this cost functional is defined in terms of the conditional state $`\omega _k`$ of the quantum system (8). However, in order to solve this optimization problem, we need to express the cost functional in a manner that facilitates the use of optimal control methods. As in the classical LEQG case, this requires the introduction of a new state, which in general is unnormalized. To define this new unnormalized state, for which we use the notation $`\widehat{\omega }`$ to distinguish such states from normalized states $`\omega `$, we need to use possibly nonlinear operators (observables) $`B`$ and (super) operators $`R`$. These nonlinear operators allow us to formulate and solve a general class of multiplicative cost optimal control problems for quantum systems666We denote the value of $`B`$ at $`\widehat{\omega }`$ by $`\widehat{\omega },B`$, extending the notation (7). The (generalized) adjoint $`R^{}`$ of $`R`$ is defined by $`\widehat{\omega },R^{}B=R\widehat{\omega },B`$ for all $`\widehat{\omega }`$ and all $`B`$..

Our risk-sensitive and multiplicative cost functionals can be defined in terms of (super) operator valued costs $`R(u)`$ that satisfy the real multiplicative homogeneity property

$$R(u)r\widehat{\omega }=rR(u)\widehat{\omega }$$
(29)
for any real number $`r`$ and any $`\widehat{\omega }`$, $`u`$. The risk-sensitive problem corresponds to particular choices of operator valued cost, Example IV.2. However, the fundamental equations in this section are valid for any operator valued cost $`R(u)`$ satisfying (29). Note that operator valued costs $`R(u)`$ are not in general quantum operations (because linearity and the inequality $`R(u)\widehat{\omega },I\widehat{\omega },I`$ need not hold in general).

###### Example IV.2

We give two examples of operator valued costs.

(i) A specific linear form for $`R(u)`$ is

$$R(u)\widehat{\omega }=\underset{c}{}Z_c(u)\widehat{\omega }Z_c(u)$$
(30)
where $`\{Z_c(u)\}`$ is a family of cost functions (section III). The adjoint $`R^{}(u)`$ acts on observables $`B`$ via

$$R^{}(u)B=\underset{c}{}Z_c(u)BZ_c(u),$$
(31)
and thereby defines a linear functional on unnormalized states by

$$\widehat{\omega },R^{}(u)B=\underset{c}{}Z_c(u)\widehat{\omega }Z_c(u),B$$
(32)
(we have written this explicitly to facilitate comparison with (34) below).

(ii) An operator valued cost $`R(u)`$ corresponding to the risk-sensitive cost (28) can be defined as follows. Let $`L(u)`$ be a cost function, and $`\mu >0`$. Then set

$$R(u)\widehat{\omega }=\frac{\widehat{\omega },e^{\mu L(u)}}{\widehat{\omega },1}\widehat{\omega }.$$
(33)
Note that $`R(u)`$ is nonlinear, but satisfies the real multiplicative homogeneity condition (29). The adjoint operator $`R^{}(u)`$ applied to an operator $`B`$ is a nonlinear functional of $`\widehat{\omega }`$ given by

$$\widehat{\omega },R^{}(u)B=\frac{\widehat{\omega },e^{\mu L(u)}}{\widehat{\omega },1}\widehat{\omega },B$$
(34)
(cf. (32) above).

The relationship between $`R(u)`$ and the risk-sensitive cost (28) will be explained in Example IV.4 below.

$`\mathrm{}`$

Given an operator valued cost $`R(u)`$, we shall find it convenient to introduce an operator $`\mathrm{\Gamma }_R(u,y)`$ defined by

$$\mathrm{\Gamma }_R(u,y)=\mathrm{\Gamma }(u,y)R(u).$$
(35)
In general, $`\mathrm{\Gamma }_R`$ is not normalized:

$$\underset{y𝐘}{}\mathrm{\Gamma }_R(u,y)\widehat{\omega },I=R(u)\widehat{\omega },I\widehat{\omega },I.$$

The operator $`\mathrm{\Gamma }_R`$ will be used to define a new state evolution as follows. Define an operator $`\mathrm{\Lambda }_{\mathrm{\Gamma },R}`$ by

$$\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u,y)\widehat{\omega }=\frac{\mathrm{\Gamma }_R(u,y)\widehat{\omega }}{p_R(y|u,\widehat{\omega })}$$
(36)
where

$$p_R(y|u,\widehat{\omega })=\frac{\mathrm{\Gamma }_R(u,y)\widehat{\omega },I}{R(u)\widehat{\omega },I}.$$
(37)
In general, the state $`\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u,y)\widehat{\omega }`$ is unnormalized. However, $`p_R(y|u,\widehat{\omega })`$ is a probability distribution, since it is easy to check that

$$\underset{y𝐘}{}p_R(y|u,\widehat{\omega })=1.$$

However, we point out that

$$\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u,y)\widehat{\omega },I=R(u)\widehat{\omega },I.$$
(38)
This unnormalized state transition operator arises in the dynamic programming equation, as we shall see below.

Associated with the operator $`\mathrm{\Lambda }_{\mathrm{\Gamma },R}`$ is the dynamics

$$\widehat{\omega }_{k+1}=\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_k,y_{k+1})\widehat{\omega }_k,$$
(39)
where $`y_{k+1}`$ is distributed according to the probability distribution $`p_R(y_{k+1}|u_k,\widehat{\omega }_k)`$ given by (37). This is a controlled Markov chain, with unnormalized states $`\widehat{\omega }_k`$. It is a modified stochastic master equation corresponding to the operator $`\mathrm{\Gamma }_R`$. Under the action of a controller $`K𝒦`$ the stochastic process $`\widehat{\omega }_k`$ is determined by (39) and $`u_k=K_k(y_{1,k})`$.

The separation structure controller in this case takes the following form. Given a function $`\widehat{𝐮}(\widehat{\omega },k)`$ and initial state $`\widehat{\omega }_0`$ we define a controller $`K_{\widehat{\omega }_0}^{\widehat{𝐮}}𝒦`$ by

$$u_k=\widehat{𝐮}(\widehat{\omega }_k,k)$$

where $`\widehat{\omega }_k`$ is given by (39), $`0kM`$, with initial condition $`\widehat{\omega }_0`$.

Let $`M>0`$ be a positive integer indicating a finite time interval $`k=0,\mathrm{},M`$. For each $`k`$, given a sequence of control values $`u_{k,M1}=u_k,\mathrm{},u_{m1}`$ and measurement values $`y_{k+1,M}=y_{k+1},\mathrm{},y_M`$, define random cost observables $`G_k`$ by the recursion

$$\begin{array}{cc}\hfill G_k& =R^{}(u_k)\mathrm{\Gamma }^{}(u_k,y_{k+1})G_{k+1},\hfill \\ \hfill G_M& =F\hfill \end{array}$$
(40)
where $`0kM1`$ and $`F`$ is a non-negative linear observable. It is evident that $`G_k`$ is real multiplicative homogeneous if $`G_{k+1}`$ is (recall (29)).

We next define the multiplicative cost functional

$$J_{\widehat{\omega },0}^\mu (K)=\underset{y_{1,M}𝐘^M}{}\widehat{\omega },G_0(K(y_{1,M})_{0,M1},y_{1,M})$$
(41)
where $`K𝒦`$ is a measurement feedback controller.

###### Lemma IV.3

The cost functional $`J_{\widehat{\omega },0}^\mu (K)`$ defined by (41) is given by the alternate expression

$$J_{\widehat{\omega },0}^\mu (K)=𝐄_{\widehat{\omega },0}^K[\widehat{\omega }_M,F]$$
(42)
where $`\widehat{\omega }_i`$, $`i=k,\mathrm{},M`$ is the solution of the recursion (39) with initial state $`\widehat{\omega }_0=\widehat{\omega }`$ under the action of the controller $`K`$.

Proof. We have

$$\begin{array}{cc}\hfill \widehat{\omega }_0,G_0& =\widehat{\omega }_0,R^{}(u_0)\mathrm{\Gamma }^{}(u_0,y_1)G_1\hfill \\ & =R(u_0)\widehat{\omega }_0,\mathrm{\Gamma }(u_0,y_1)^{}G_1\hfill \\ & =\mathrm{\Gamma }(u_0,y_1)R(u_0)\widehat{\omega }_0,G_1\hfill \\ & =\widehat{\omega }_1,G_1)p_R(y_1|u_0,\widehat{\omega }_0)\hfill \end{array}$$

where $`\widehat{\omega }_1=\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_0,y_1)\widehat{\omega }_0`$ and $`p_R(y_1|u_0,\widehat{\omega }_0)`$ is given by (37). Iterating in this way we see that (41) and (42) are equivalent. These properties use the real multiplicative homogeneity property of $`G_k`$. $`\mathrm{}`$

###### Example IV.4

(Continuation of Example IV.2 (ii).) We now show that when $`R(u)`$ is given by (33), and

$$F=e^{\mu N},$$

where $`N`$ is a non-negative linear observable, the multiplicative cost functional $`J_{\widehat{\omega },0}^\mu (K)`$ defined by (41) equals the risk-sensitive cost functional (28).

Proceeding as in the proof of Lemma IV.3 we have

$$\begin{array}{cc}& \widehat{\omega }_0,G_0\hfill \\ & =\widehat{\omega }_0,R^{}(u_0)\mathrm{\Gamma }^{}(u_0,y_1)G_1\hfill \\ & =\mathrm{\Gamma }(u_0,y_1)R(u_0)\widehat{\omega }_0,G_1\hfill \\ & =\mathrm{\Gamma }(u_0,y_1)\widehat{\omega }_0,G_1\frac{\widehat{\omega }_0,e^{\mu L(u_0)}}{\widehat{\omega }_0,1}\hfill \\ & =\frac{\mathrm{\Gamma }(u_0,y_1)\widehat{\omega }_0,G_1/\widehat{\omega }_0,1}{\mathrm{\Gamma }(u_0,y_1)\widehat{\omega }_0,1/\widehat{\omega }_0,1}\widehat{\omega }_0,e^{\mu L(u_0)}.\hfill \\ & .\frac{\mathrm{\Gamma }(u_0,y_1)\widehat{\omega }_0,1}{\widehat{\omega }_0,1}\hfill \\ & =\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\overline{\widehat{\omega }}_0,G_1\widehat{\omega }_0,e^{\mu L(u_0)}p(y_1|u_0,\overline{\widehat{\omega }}_0)\hfill \end{array}$$

where $`\overline{\widehat{\omega }}_0=\widehat{\omega }_0/\widehat{\omega }_0,1`$, $`\mathrm{\Lambda }_\mathrm{\Gamma }(u,y)`$ is defined by (9) and $`p(y|u,\omega )`$ is defined by (6). Now if $`\widehat{\omega }_0=\omega _0`$ is normalized, with $`\omega _0,1=1`$, then we have shown that

$$\begin{array}{cc}& \omega _0,G_0\hfill \\ & =\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\omega _0,G_1\omega _0,e^{\mu L(u_0)}p(y_1|u_0,\overline{\omega }_0)\hfill \\ & =\omega _1,G_1\omega _0,e^{\mu L(u_0)}p(y_1|u_0,\omega _0)\hfill \end{array}$$

where $`\omega _1=\mathrm{\Lambda }_\mathrm{\Gamma }(u_0,y_1)\omega _0`$ is the normalized state evolving according to the quantum system model (8). Note that $`\omega _1,1=1`$. Continuing in this way we see that (41) equals the risk-sensitive cost functional (28), using Lemma IV.3.

It can also be checked that $`\omega _k`$ and $`\widehat{\omega }_k`$ are related simply via

$$\widehat{\omega }_k=\underset{i=0}{\overset{k1}{}}\omega _i,e^{\mu L(u_i)}\omega _k.$$
(43)
$`\mathrm{}`$

To solve the optimal control problem for the cost functional (41), we define the cost to go

$$\begin{array}{c}J_{\widehat{\omega },k}^\mu (K)\hfill \\ =\underset{y_{k+1,M}𝐘^{Mk}}{}\widehat{\omega },G_k(K(y_{k+1,M})_{k,M1},y_{k+1,M})\hfill \end{array}$$
(44)
and the corresponding dynamic programming equation

$$\begin{array}{cc}\hfill W(\widehat{\omega },k)& =\underset{u𝐔}{inf}\{\underset{y𝐘}{}W(\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u,y)\widehat{\omega },k+1)\hfill \\ & .p_R(y|u,\widehat{\omega })\},\hfill \\ \hfill W(\widehat{\omega },M)& =\widehat{\omega },F\hfill \end{array}$$
(45)
where $`0kM1`$.

###### Theorem IV.5

Let $`W(\widehat{\omega },k)`$, $`0kM`$, be the solution of the dynamic programming equation (45). (i) Then for any $`K𝒦`$ we have

$$W(\widehat{\omega },k)J_{\widehat{\omega },k}^\mu (K)$$
(46)
(ii) Assume in addition that the minimizer

$$\begin{array}{c}\widehat{𝐮}^{}(\widehat{\omega },k)\hfill \\ \underset{u𝐔}{argmin}\{\underset{y𝐘}{}W(\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u,y)\widehat{\omega },k+1)p_R(y|u,\widehat{\omega })\}\hfill \end{array}$$
(47)
exists for all $`\widehat{\omega }`$, $`0kM1`$. Then the separation structure controller $`K_{\widehat{\omega }_0}^{\widehat{𝐮}^{}}`$ defined by (47) is optimal for problem (41), i.e. $`J_{\widehat{\omega }_0,0}^\mu (K)J_{\widehat{\omega }_0,0}^\mu (K_{\widehat{\omega }_0}^{\widehat{𝐮}^{}})`$ for all $`K𝒦`$.

Proof. We prove part (i) by induction. Let $`K𝒦`$. For $`k=M`$, we have

$$W(\widehat{\omega },M)=\widehat{\omega },F=J_{\widehat{\omega },M}^\mu (K)$$

so (46) holds for $`k=M`$. Next, we assume (46) holds for $`k+1`$, i.e.

$$W(\widehat{\omega },k+1)J_{\widehat{\omega },k+1}^\mu (K)$$
(48)
Now by (45), (48) and (37)

$$\begin{array}{cc}& W(\widehat{\omega },k)\hfill \\ & \underset{y_{k+1}𝐘}{}W(\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_k,y_{k+1})\widehat{\omega },k+1)p_R(y_{k+1}|u_k,\widehat{\omega })\hfill \\ & \underset{y_{k+1}𝐘}{}J_{\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_k,y_{k+1})\widehat{\omega },k+1}^\mu (K)p_R(y_{k+1}|u_k,\widehat{\omega })\hfill \\ & =\underset{y_{k+1}𝐘}{}\underset{y_{k+2,M}𝐘^{M(k+1)}}{}\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_k,y_{k+1})\widehat{\omega },G_{k+1}.\hfill \\ & .p_R(y_{k+1}|u_k,\widehat{\omega })\hfill \\ & =\underset{y_{k+1}𝐘}{}\underset{y_{k+2,M}𝐘^{M(k+1)}}{}\widehat{\omega },\mathrm{\Gamma }_R^{}(u_k,y_{k+1})G_{k+1}\hfill \\ & =J_{\widehat{\omega },k}^\mu (K)\hfill \end{array}$$

as required.

Part (ii) follows from the proof of part (i), with $`k=0`$, since at every step we have equality and so

$$W(\widehat{\omega }_0,0)=J_{\widehat{\omega }_0,0}^\mu (K_{\widehat{\omega }_0}^{\widehat{𝐮}^{}})$$

Hence $`J_{\widehat{\omega }_0,0}^\mu (K)J_{\widehat{\omega }_0,0}^\mu (K_{\widehat{\omega }_0}^{\widehat{𝐮}^{}})`$ for all $`K𝒦`$. The real multiplicative homogeneity property of $`G_k`$ has been used here also. $`\mathrm{}`$

###### Remark IV.6

Note that the optimal multiplicative cost/risk-sensitive controller $`K_{\widehat{\omega }_0}^{\widehat{𝐮}^{}}`$ determined by (47) feeds back the unnormalized conditional state $`\widehat{\omega }_k`$, given by the modified SME (39), Figure 7. This means that from the point of view of optimal risk-sensitive or multiplicative control, the best thing to do involves use of a dynamical quantity that not only contains knowledge (as measured by the controller) of the quantum system, but also contains information about the purpose of the controller. This should be contrasted with the risk-neutral case, section III. Note in particular that the modified SME (39) is no longer the optimal filter from the point of view of seeking the best estimate of observables (c.f. Remark III.4). Further, the concept of dual control (Remark III.5) has greater weight here, since the cost $`R(u)`$ appears explicitly in the controller dynamics (39)—while the optimal multiplicative cost/risk-sensitive controller has a separation structure, in the sense of a decomposition into a dynamical filter part and static control part, the task of estimation is not separated from the task of control, Figure 7. We emphasize that the multiplicative cost/risk-sensitive conditional state is defined only in the context of these specific control objectives, where they are used in specific feedback situations. $`\mathrm{}`$

###### Example IV.7

(Two-state system with feedback, Example III.3 continued.) We now consider the risk-sensitive optimal control problem for the two-state example, with operator valued cost $`R(u)`$ defined by (33) where $`L(u)`$ is given by (19). If we write the density matrix as (15), then

$$R(u)\widehat{\omega }=\frac{e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}e^{\mu c(u)}\left(\begin{array}{cc}\widehat{\omega }_{11}& \widehat{\omega }_{12}\\ \widehat{\omega }_{12}^{}& \widehat{\omega }_{22}\end{array}\right).$$

The risk-sensitive controlled transfer operators $`\mathrm{\Gamma }_R(u,y)`$ are defined by

$$\begin{array}{cc}\hfill \mathrm{\Gamma }_R(u,y)\widehat{\omega }& =\mathrm{\Gamma }(u,y)R(u)\widehat{\omega }\hfill \\ & =\frac{\widehat{\omega },e^{\mu L(u)}}{\widehat{\omega },1}\mathrm{\Gamma }(u,y)\widehat{\omega }.\hfill \end{array}$$

Explicitly, we have

$$\begin{array}{cc}\hfill \mathrm{\Gamma }_R(0,1)\widehat{\omega }& =\frac{e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}\left(\begin{array}{cc}(1\alpha )\widehat{\omega }_{11}& 0\\ 0& \alpha \widehat{\omega }_{22}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }_R(0,1)\widehat{\omega }& =\frac{e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}\left(\begin{array}{cc}\alpha \widehat{\omega }_{11}& 0\\ 0& (1\alpha )\widehat{\omega }_{22}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }_R(1,1)\widehat{\omega }& =\frac{e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}e^{\mu p}\left(\begin{array}{cc}(1\alpha )\widehat{\omega }_{22}& 0\\ 0& \alpha \widehat{\omega }_{11}\end{array}\right)\hfill \\ \hfill \mathrm{\Gamma }_R(1,1)\widehat{\omega }& =\frac{e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}e^{\mu p}\left(\begin{array}{cc}\alpha \widehat{\omega }_{22}& 0\\ 0& (1\alpha )\widehat{\omega }_{11}\end{array}\right).\hfill \end{array}$$

The dynamic programming equation (45) is solved and the optimal feedback controls are found as follows. First, for $`k=M=2`$ we have

$$W(\widehat{\omega },2)=\widehat{\omega },e^{\mu X^2}=e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}$$

and then for $`k=1`$

$$W(\widehat{\omega },1)=\mathrm{min}[W_0(\widehat{\omega },1),W_0(\widehat{\omega },1)]$$

where $`W_0(\widehat{\omega },1)`$ and $`W_1(\widehat{\omega },1)`$ are given in Appendix A, and

$$\widehat{𝐮}^{}(\omega ,1)=\{\begin{array}{cc}\hfill 0& \text{if}W_0(\omega ,1)W_1(\omega ,1)\hfill \\ \hfill 1& \text{if}W_0(\omega ,1)>W_1(\omega ,1).\hfill \end{array}$$

Next, for $`k=0`$,

$$W(\widehat{\omega },0)=\mathrm{min}[W_0(\widehat{\omega },0),W_0(\widehat{\omega },0)]$$

where $`W_0(\widehat{\omega },0)`$ and $`W_1(\widehat{\omega },0)`$ are given in Appendix A, and

$$\widehat{𝐮}^{}(\omega ,0)=\{\begin{array}{cc}\hfill 0& \text{if}W_0(\omega ,0)W_1(\omega ,0)\hfill \\ \hfill 1& \text{if}W_0(\omega ,0)>W_1(\omega ,0).\hfill \end{array}$$

The optimal feedback controller is given by

$$u_0=K_{\widehat{\omega }_0,0}^{\widehat{𝐮}^{}}=\widehat{𝐮}^{}(\widehat{\omega }_0,0),u_1=K_{\widehat{\omega }_0,1}^{\widehat{𝐮}^{}}(y_1)=\widehat{𝐮}^{}(\omega _1,1)$$

where $`\widehat{\omega }_1=\mathrm{\Lambda }_{\mathrm{\Gamma },R}(u_0,y_1)\widehat{\omega }_0`$. Again, we see the separation structure, where here the control $`u_1`$ depends on $`y_1`$ through the unnormalized conditional state $`\widehat{\omega }_1`$. A physical implementation of this controller is shown in Figure 8.

To see the effect of the risk-sensitive controller, consider the initial state $`\widehat{\omega }_0=\omega _0`$ given by (17), and parameter values $`\alpha =0.25`$, $`\mu =2`$. We find that $`W_0(\widehat{\omega }_0,0)=W_1(\widehat{\omega }_0,0)`$, and hence we take $`u_0=\widehat{𝐮}^{}(\widehat{\omega }_0,0)=0`$.

If $`y_1=1`$ is measured, we find that

$$\begin{array}{cc}\hfill \widehat{\omega }_1& =\left(\begin{array}{cc}3.1459& 0\\ 0& 1.04863\end{array}\right),\hfill \\ \hfill \omega _1& =\left(\begin{array}{cc}0.75& 0\\ 0& 0.25\end{array}\right)\text{with prob.}0.5,\hfill \end{array}$$

and

$$u_1=\widehat{𝐮}^{}(\widehat{\omega }_1,1)=\{\begin{array}{cc}\hfill 1& \text{if}0p0.4\hfill \\ \hfill 0& \text{if}p>0.4\hfill \end{array}$$
(49)
If, on the other hand, $`y_1=1`$ is measured, we find that

$$\begin{array}{cc}\hfill \widehat{\omega }_1& =\left(\begin{array}{cc}1.04863& 0\\ 0& 3.1459\end{array}\right),\hfill \\ \hfill \omega _1& =\left(\begin{array}{cc}0.25& 0\\ 0& 0.75\end{array}\right)\text{with prob.}0.5,\hfill \end{array}$$

and

$$u_1=\widehat{𝐮}^{}(\widehat{\omega }_1,1)=0$$

for any value of $`p`$. When the control cost $`p=0.2`$, the final (non-selective) state is given by

$$\rho _2=\left(\begin{array}{cc}0.25& 0\\ 0& 0.75\end{array}\right)=0.25|11|+0.75|11|$$

This state does not equal the desired pure state $`|11|`$, a reflection of the level of measurement uncertainty $`\alpha =0.25`$ and the presence of a non-zero control penalty.

To compare with the risk-neutral version of this problem, we find that the threshold in (49) for the risk-neutral problem is $`p=0.75`$. This means that for a larger range of values of the control cost $`p`$, the risk-neutral controller will be active, i.e. select $`u=1`$ than is the case for the risk-sensitive controller. This is consistent with the description of the example in FGM94 ; CM95  where the risk-neutral controller is more aggressive than the risk-sensitive controller. $`\mathrm{}`$

We conclude with an example which indicates the likely robustness properties of the risk-sensitive controller and the relationship between the risk-neutral and risk-sensitive problems.

###### Example IV.8

We consider the risk-sensitive cost functional (28), where where the operator valued cost $`R(u)`$ has the form (33), and $`F=e^{\mu N}`$.

Robustness. To describe the robustness properties of the risk-sensitive controller, we follow DJP00  and make use of the following general convex duality formula (see, e.g. (DE97, , Chapter 1.4)):

$$\mathrm{log}𝐄_𝐏[e^f]=\underset{𝐐}{sup}\{𝐄_𝐐[f]RE(𝐐𝐏)\}$$
(50)
where $`𝐏`$ and $`𝐐`$ are probability distributions777We require that $`𝐐`$ is absolutely continuous with respect to $`𝐏`$., and where the relative entropy is defined by (see, e.g., (NC00, , Chapter 11))

$$RE(𝐐𝐏)=𝐄_𝐐[\mathrm{log}\frac{d𝐐}{d𝐏}].$$

To apply formula (50), we proceed as follows. Let $`\mathrm{\Gamma }_{nom}`$ be the nominal operator used for design of the optimal risk-sensitive controller, here denoted $`\widehat{K}_{nom}^{}`$. Together, $`\mathrm{\Gamma }_{nom}`$ and $`\widehat{K}_{nom}^{}`$ determine a probability distribution, here denoted $`𝐏_{nom}`$. In reality, the nominal $`\mathrm{\Gamma }_{nom}`$ need not equal the operator for the “true” system, denoted $`\mathrm{\Gamma }_{true}`$. The controller $`\widehat{K}_{nom}^{}`$ is applied to the true system, resulting in a probability distribution $`𝐏_{true}`$888We assume that the true distribution $`𝐏_{true}`$ is absolutely continuous with respect to the nominal distribution $`𝐏_{nom}`$..

We write $`\mu =1/\gamma ^2`$, and apply (50) to obtain the following inequality ($`𝐏=𝐏_{nom}`$, $`𝐐=𝐏_{true}`$):

$$\begin{array}{c}\gamma ^2\mathrm{log}𝐄_{𝐏_{true}}[_{k=0}^{M1}\omega _k,e^{\mu L(u_k)}\omega _M,e^{\mu N}]\hfill \\ 𝐄_{𝐏_{true}}[\gamma ^2\mathrm{log}(_{k=0}^{M1}\omega _k,e^{\mu L(u_k)}\omega _M,e^{\mu N})]\hfill \\ \gamma ^2RE(𝐏_{true}𝐏_{nom})\hfill \\ =𝐄_{𝐏_{true}}[\gamma ^2(_{k=0}^{M1}\mathrm{log}\omega _k,e^{\mu L(u_k)}\hfill \\ +\mathrm{log}\omega _M,e^{\mu N})]\hfill \\ \gamma ^2RE(𝐏_{true}𝐏_{nom})\hfill \\ 𝐄_{𝐏_{true}}[_{k=0}^{M1}\omega _k,L(u_k)+\omega _M,N)]\hfill \\ \gamma ^2RE(𝐏_{true}𝐏_{nom})\hfill \end{array}$$

This implies the important bound:

$$\begin{array}{c}J_{𝐏_{true}}^{rn}(\overline{K}_{nom}^{})\hfill \\ \gamma ^2\mathrm{log}J_{𝐏_{nom}}^{rs,\gamma ^2}(\widehat{K}_{nom}^{})+\gamma ^2RE(𝐏_{true}𝐏_{nom})\hfill \end{array}$$
(51)
The LHS of (51) is the risk-neutral cost criterion (20), evaluated using the true system model $`𝐏_{true}`$ and the controller $`\widehat{K}_{nom}^{}`$ designed using the nominal model $`𝐏_{nom}`$. Inequality (51) bounds this cost by two terms, the first term is related to the optimal risk-sensitive cost (28), while the second is the relative entropy term, which is a measure of the “distance” between the true and nominal systems. The number $`\gamma ^2=1/\mu >0`$ is a “robustness gain” parameter, which we would like to be as small as possible for maximum robustness, as in $`H^{\mathrm{}}`$ robust control, GL95 , where the relative entropy term is a measure of the “energy” in the disturbance or uncertainty. This shows that the risk-sensitive controller enjoys good performance, as measured by the risk-neutral criterion, under nominal conditions $`(𝐏_{true}=𝐏_{nom})`$, and acceptable performance in other than nominal conditions $`(𝐏_{true}𝐏_{nom})`$, as implied by the bound. In summary, risk-sensitive controllers enjoy enhanced robustness (recall Remark III.6).

Relationship between the risk-neutral and risk-sensitive value functions. We indicate briefly how the results of (JBE94, , Theorem 5.5) apply in the present context. Indeed, the reader may check that for small $`\mu >0`$ one has

$$\frac{1}{\mu }\mathrm{log}\frac{\widehat{\omega },\mathrm{exp}(\mu N)}{\widehat{\omega },1}\frac{\widehat{\omega },N}{\widehat{\omega },1}$$

This suggests the relation

$$\underset{\mu 0}{lim}\frac{1}{\mu }\mathrm{log}\frac{W(\widehat{\omega },k)}{\widehat{\omega },1}=\frac{V(\widehat{\omega },k)}{\widehat{\omega },1},$$
(52)
which says that a logarithmic risk-sensitive optimal cost tends to the optimal risk-neutral cost as the parameter $`\mu 0`$, as might be expected. $`\mathrm{}`$

## Appendix A Formulas for the Two-State System with Feedback Example

The following quantities were used in the solution of the risk-neutral problem, Example III.3:

$$V_0(\omega ,1)=\omega _{11},V_1(\omega ,1)=\omega _{22}+p$$

$$\begin{array}{c}V_0(\omega ,0)=\omega _{11}+\mathrm{min}[\alpha \omega _{11},p+\omega _{22}\alpha \omega _{22}]\hfill \\ +\mathrm{min}[\omega _{11}\alpha \omega _{11},p+\alpha \omega _{22}]\hfill \end{array}$$

$$\begin{array}{c}V_1(\omega ,0)=p+\alpha \omega _{11}+\omega _{22}\alpha \omega _{22}\hfill \\ +\mathrm{min}[\alpha \omega _{11},p+\omega _{22}\alpha \omega _{22}]\hfill \\ +\mathrm{min}[p+\alpha \omega _{11},\omega _{22}\alpha \omega _{22}]\hfill \end{array}$$

The following quantities were used in the solution of the risk-sensitive problem, Example IV.7:

$$W_0(\widehat{\omega },1)=\frac{(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})^2}{\widehat{\omega }_{11}+\widehat{\omega }_{22}},W_1(\widehat{\omega },1)=\frac{e^{\mu p}(\widehat{\omega }_{11}\widehat{\omega }_{22}+e^{2\mu }\widehat{\omega }_{11}\widehat{\omega }_{22}+e^\mu (\widehat{\omega }_{11}^{}{}_{}{}^{2}+\widehat{\omega }_{22}^{}{}_{}{}^{2}))}{\widehat{\omega }_{11}+\widehat{\omega }_{22}}$$

$$\begin{array}{cc}\hfill W_0(\widehat{\omega },0)& =\mathrm{min}[(\frac{(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})((1+\alpha )e^\mu \widehat{\omega }_{11}\alpha \widehat{\omega }_{22})^2}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})((1+\alpha )\widehat{\omega }_{11}\alpha \widehat{\omega }_{22})}),\hfill \\ & (\frac{e^{\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(((1+\alpha )\alpha \widehat{\omega }_{11}\widehat{\omega }_{22})(1+\alpha )\alpha e^{2\mu }\widehat{\omega }_{11}\widehat{\omega }_{22}+e^\mu ((1+\alpha )^2\widehat{\omega }_{11}^{}{}_{}{}^{2}+\alpha ^2\widehat{\omega }_{22}^{}{}_{}{}^{2}))}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})((1+\alpha )\widehat{\omega }_{11}\alpha \widehat{\omega }_{22})})]\hfill \\ & +\mathrm{min}([\frac{(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(\alpha e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22}\alpha \widehat{\omega }_{22})^2}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})(\alpha (\widehat{\omega }_{11}\widehat{\omega }_{22})+\widehat{\omega }_{22})},\hfill \\ & \frac{e^{\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(e^\mu \widehat{\omega }_{22}^{}{}_{}{}^{2}+\alpha \widehat{\omega }_{22}(\widehat{\omega }_{11}+e^{2\mu }\widehat{\omega }_{11}2e^\mu \widehat{\omega }_{22})+\alpha ^2((\widehat{\omega }_{11}\widehat{\omega }_{22})e^{2\mu }\widehat{\omega }_{11}\widehat{\omega }_{22}+e^\mu (\widehat{\omega }_{11}^{}{}_{}{}^{2}+\widehat{\omega }_{22}^{}{}_{}{}^{2})))}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})(\alpha (\widehat{\omega }_{11}\widehat{\omega }_{22})+\widehat{\omega }_{22})}]\hfill \end{array}$$

$$\begin{array}{cc}\hfill W_1(\widehat{\omega },0)& =\mathrm{min}[(\frac{e^{\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(\widehat{\omega }_{11}\alpha \widehat{\omega }_{11}+\alpha e^\mu \widehat{\omega }_{22})^2}{(wr11+\widehat{\omega }_{22})((1+\alpha )\widehat{\omega }_{11}\alpha \widehat{\omega }_{22})}),\hfill \\ & (\frac{e^{2\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(((1+\alpha )\alpha \widehat{\omega }_{11}\widehat{\omega }_{22})(1+\alpha )\alpha e^{2\mu }\widehat{\omega }_{11}\widehat{\omega }_{22}+e^\mu ((1+\alpha )^2\widehat{\omega }_{11}^{}{}_{}{}^{2}+\alpha ^2\widehat{\omega }_{22}^{}{}_{}{}^{2}))}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})((1+\alpha )\widehat{\omega }_{11}\alpha \widehat{\omega }_{22})})]\hfill \\ & +\mathrm{min}[\frac{e^{\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(e^\mu \widehat{\omega }_{22}+\alpha (\widehat{\omega }_{11}e^\mu \widehat{\omega }_{22}))^2}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})(\alpha (\widehat{\omega }_{11}\widehat{\omega }_{22})+\widehat{\omega }_{22})},\hfill \\ & \frac{e^{2\mu p}(e^\mu \widehat{\omega }_{11}+\widehat{\omega }_{22})(e^\mu \widehat{\omega }_{22}^{}{}_{}{}^{2}+\alpha \widehat{\omega }_{22}(\widehat{\omega }_{11}+e^{2\mu }\widehat{\omega }_{11}2e^\mu \widehat{\omega }_{22})+\alpha ^2((\widehat{\omega }_{11}\widehat{\omega }_{22})e^{2\mu }\widehat{\omega }_{11}\widehat{\omega }_{22}+e^\mu (\widehat{\omega }_{11}^{}{}_{}{}^{2}+\widehat{\omega }_{22}^{}{}_{}{}^{2})))}{(\widehat{\omega }_{11}+\widehat{\omega }_{22})(\alpha (\widehat{\omega }_{11}\widehat{\omega }_{22})+\widehat{\omega }_{22})}]\hfill \end{array}$$


format: text

text:
# Black Hole Production at a TeV11footnote 1Invited talk at the 12th Workshop on General Relativity and Gravitation, Komaba Campus, University of Tokyo, Tokyo, Japan November 25-28, 2002.

## 1 Introduction

Let us start by asking one of the most basic questions in theoretical physics: What is the fundamental scale for Quantum Gravity? Answer:We don’t know. It is only quite recently that we have become fully aware of this. The reason for our ignorance is quite simple: we do not know how to formulate the correct theory of quantum gravity. What we do know is the following: the fundamental energy scale $`M_{}`$ has to be larger than a TeV, for otherwise we should have seen effects of quantum gravity at LEP and the Tevatron444There are ways of sidestepping even this lower bound , but I will not discuss these scenarios here.. We also know that $`M_{}`$ can be as large as, but not larger than, the familiar Planck scale $`M_{(4)}10^{19}`$ GeV, since at this energy four-dimensional gravity becomes strongly coupled and quantum effects surely must become important. The phrase four-dimensional in the last statement betrays already the mechanism that allows $`M_{}`$ to be smaller than $`M_{(4)}`$ . The introduction of large extra dimensions dilutes gravity in an internal $`n`$-dimensional volume $`V_n=L^n`$, and $`M_{(4)}`$ is revealed as an effective coupling

$$M_{(4)}^2=M_{}^{n+2}L^n,$$
(1)
describing the strength of gravity only at distances larger than the length scale of the additional dimensions.

The scenario is completed by a mechanism that prevents all particles and interactions, other than gravity, to spread out into these large extra dimensions: they are confined inside a brane, of thickness $`\begin{array}{c}<\hfill \\ \hfill \end{array}`$ TeV-1, localized in the internal space. By making $`L^n`$ sufficiently large to allow $`M_{}`$ to be near the TeV scale, one achieves a reformulation of the hierarchy problem.

The great interest in these models has been prompted by the fact that, with $`M_{}`$ as low as a TeV, one obtains spectacular new physics in future colliders such as the LHC. The signatures due to graviton production and exchange have been thoroughly studied (see e.g. ). In this talk I will briefly review some aspects of a more striking signature of TeV quantum gravity/strings and large extra dimensions: the production of exotic objects such as black holes , and their stringy precursors, the string resonances at large masses which can be dubbed ‘string balls’ . Both these objects have in common their large mass (larger than fundamental mass scale), and the fact that they have a large degeneracy of internal states. The latter is crucial to their large production cross section.

There are already a number of reviews on this subject  to which the reader is referred to for other details, for discussion of topics that I mentioned in my talk but are not included here such as black hole production in cosmic rays , and for extensive lists of references. Instead, I will mostly focus on aspects that have been less prominent in former discussions in the literature.

## 2 Black holes on branes

Finding the exact description of a black hole localized on a three-brane, whether the bulk spacetime is compact (as in ADD scenarios) or non-compact (as in the RS2 model), is an outstanding problem in General Relativity. However, the problem can be sidestepped for a black hole formed at a typical collider energy in the range of $`O(10)`$ TeV. Such a black hole has a size $`10^{15}`$ mm, which is several orders of magnitude smaller than the size that the extra dimensions can presumably have (e.g., for $`n=7`$ and $`M_{}O(`$TeV$`)`$ we would have $`L`$ fermi). Therefore, at distances smaller than $`L`$ we can neglect all finite-size effects from the compact dimensions, and approximate the metric for a neutral, non-rotating black hole by the Tangherlini-Schwarzschild solution in $`4+n`$ dimensions. We want this black hole to be stuck to the brane, in such a manner that, at distances $`rL`$ the metric induced on the brane is

$$ds_{brane}^2\left(1\frac{\mu }{r^{1+n}}\right)dt^2+\frac{dr^2}{\left(1\frac{\mu }{r^{1+n}}\right)}+r^2d\mathrm{\Omega }_{(2)}^2.$$
(2)
All the interesting physics (the formation and evaporation of the black hole) happens in a region within a few horizon radii, so this approximate geometry will be enough for our purposes. Note that this is not a solution of the Einstein equations in four dimensions: the additional modes of gravity in the bulk (Kaluza-Klein modes) are not zero in the region near the black hole horizon. It can also be argued that the mass of the black hole as measured from the $`4+n`$ dimensional point of view is the same as the one that a four-dimensional observer, probing distances $`rL`$, would measure .

## 3 Scattering at $`E>M_{}`$ and Black Hole Formation

Given our ignorance about the correct theory of quantum gravity, one might think that we cannot say anything at all about scattering at energies beyond the fundamental Planck scale $`M_{}`$. This is not quite correct.555Ideas similar or related to those in this section have been discussed previously in, e.g., . While there is certainly a gap in our ability to perform calculations at energies near the fundamental scale of quantum gravity, $`EM_{}`$, a number of generic arguments based on a crucial property of General Relativity, namely the appearance of horizons, allow us to argue for a description of some features of scattering in the transplanckian energy regime. There are certainly effects that, at present, we do not even have a proper theory to calculate, but in many instances an asymptotic observer will be causally cut off from them. The crucial point is cosmic censorship, which can be taken to say, quite liberally, that the extreme high energy regime will be shrouded by a horizon. Two particles scattering at transplanckian center-of-mass energy, $`\sqrt{s}`$, and small impact parameter $`b`$, will produce a large concentration of energy in a small volume, and therefore will induce a large spacetime curvature, larger than the fundamental scale. To a low-energy observer, this will be equivalent to a curvature singularity. Then, if cosmic censorship holds, a horizon will appear, hiding these higher-energy effects from the observers outside.

This generic argument assumes that in the collision of two transplanckian particles their energy actually gets concentrated in a small volume. One may worry that the emission of a large fraction of energy into initial state radiation might prevent this. However, this does not seem to be the case. The analyses of  show that trapped surfaces do form in the collision of classical transplanckian particles. Quite generically, these trapped surfaces appear before the colliding particles are at a distance where quantum gravity corrections (e.g. higher-dimension operators in the low energy effective action) must be taken into account. Once the trapped surfaces arise, a singularity will inevitably develop, and a horizon is expected to censor it out. The area of the apparent horizon (the outer envelope of the trapped surfaces), which cannot be larger than the area of the event horizon, is found to be smaller than, but still close to the area of a black hole of mass equal to $`\sqrt{s}`$. Hence follows a (dimension-dependent) upper bound on the fraction of gravitational radiation that is classically produced.666Still, the question remains of whether the radiation of quantum origin is also small. I thank Riccardo Rattazzi for discussions on this issue.

Low energy effective field theory predicts the failure of General Relativity at energies near or above the Planck scale by the effect of higher-dimension operators in the low energy effective action becoming as important as the Einstein-Hilbert term. What the argument above says is that the effects of these operators will be largely confined inside a black hole horizon. An alternative view, using a spacetime description, would be the following: a region of large curvature can be enclosed by a surface where all curvatures are moderate; say, below a given cutoff curvature radius. This region can be cutoff and then replaced by boundary conditions at its surface. The main assumption, analogous to the locality of counterterms in the low energy effective action, is that the specification of these boundary conditions is sufficient to encode all the effects that the physics at higher curvatures would have on the lower curvature regions. The situation changes if a horizon appears, since it not only excises the large curvature region, but the horizon itself also provides a set of boundary conditions (i.e., thermal) that is largely independent of the details of the large curvatures inside the horizon777The extent to which the properties of the horizon are so universal and independent of the physics at short distances is closely related to the black hole no-hair conjecture, and to the problem of information loss..

These considerations reveal an essential property of quantum gravity: its softness at high energies. The higher the energy of the collision, the larger the horizon that forms, and therefore the less important that the quantum gravity corrections will be for an observer outside the horizon. So gravitational scattering at very high energies becomes softer and softer. Ultimately, the outcome of the scattering will be the Hawking radiation emitted in the decay of the black hole, whose wavelength is comparable to the black hole size, and therefore larger for higher energy collisions. This softness is also a property that can be observed in the eikonalized gravitational scattering amplitudes, in which one can have a large momentum transfer, but built out of a very large number of small momentum transfers, so at the end one does not probe the very shortest distances . It is also strongly reminiscent of the softness of strings in the regime of deep inelastic scattering. The reasons for it are at first sight quite different. One can see the parallels both in the Regge regime and in the deep inelastic scattering regime, and in fact the black hole/string correspondence principle suggest such a connection, which to our knowledge has not been discussed in detail in the literature. The correspondence principle also allows us to use string theory to perform a simple calculation that leads to the correct estimate for the black hole production cross section. This will be discussed in Sec. 5 below.

At transplanckian energies one must go beyond perturbation theory, but it is the classical gravitational dynamics that dominates the evolution of the system888Semiclassical, if one also includes the decay via Hawking radiation.. This is apparent both in the case where a black hole forms, and also in the eikonal analysis of the scattering. It can be interpreted as saying that one must consider diagrams with an arbitrary number of vertex insertions, but where graviton loops are absent; matter lines in the diagrams are taken to be on-shell.

A full calculation of the production cross section, using classical General Relativity, is a very complicated task. A simple estimate can be obtained by taking into account the above-mentioned bounds on initial state radiation, and other factors, such as the fact that the capture cross section of a black hole is actually larger than its area, which may raise the cross section. For practical purposes, it seems reasonable to estimate the black hole production cross section as the geometric cross section

$$\sigma \pi R_H^2\frac{1}{M_{}}\left(\frac{\sqrt{s}}{M_{}}\right)^{\frac{2}{n+1}}.$$
(3)
Since this cross section does not contain any small numbers that might suppress it, at energies above $`M_{}`$ TeV it leads to enormous cross sections, on the order of TeV$`{}_{}{}^{2}O(100)`$ pb. More careful estimates result in production rates at the LHC as large as a black hole per second .

## 4 Black hole decay

Once the black hole forms, it will decay very rapidly due to Hawking evaporation. But, where is this radiation emitted? This is a crucial point, since if the radiation were emitted into the KK modes of the bulk graviton it would escape our detectors and we would not be able to distinguish black hole formation from other missing energy events. In fact, a simple argument appears to lead precisely to this conclusion: Since the extra dimensions are assumed to be much larger than the black hole, from the point of view of a four-dimensional observer there is an enormous number of light KK modes, of mass smaller than the Hawking temperature. Emission into these modes would then overwhelm the radiation of the Standard Model particles that are emitted along the brane. Viewed another way, the phase space available for emission into the bulk is much larger than for emission along the brane, making virtually zero the fraction of energy that could possibly be detected.

Even if it was clear from early on that TeV-gravity scenarios greatly enhance black hole production , these arguments prevented further discussion of their possible detection at colliders. However, in  a number of counterarguments were given to conclude that the outcome of the evaporation is emitted mostly along the brane and therefore can actually be detected. This opened the door to the studies of the actual phenomenology of the process .

Instead of repeating the arguments of , here I will present an ellaboration of an equivalent but slightly different argument, originally due to L. Susskind.

The (erroneous) argument based on the fact that the number of KK modes is much larger than the number of brane modes assumes that Hawking radiation is emitted with roughly the same probability into all these KK modes and into the brane modes. Let us revisit this line of reasoning, but instead of labelling the KK modes by the momentum in the extra dimensions, in a plane-wave basis decomposition, we will decompose the modes in a basis of spherical harmonics. Then we classify the modes according to their partial-wave number $`\mathrm{}`$. Viewed this way, the fact that there are many more modes coming from the $`4+n`$-dimensional bulk than from the four-dimensional brane is reflected in the fact that, for large $`\mathrm{}`$, the degeneracy of a mode of given $`\mathrm{}`$ grows in the former case as $`\mathrm{}^{n+1}`$, much faster than the degeneracy on the four-dimensional brane, which is well-known to be $`2\mathrm{}+1`$, i.e., $`\mathrm{}`$. Imagine now a hot object of characteristic size $`R`$, which is radiating thermally at a typical wavelength $`\lambda R`$. There will be radiation into modes of partial-wave number up to the large value $`\mathrm{}_{max}R/\lambda `$. Since there are many more of these modes in higher dimensions, much more radiation will go into the bulk than into the brane.

However, a black hole is not this sort of object. The typical wavelength of Hawking radiation is actually on the order of, or a little larger than, the black hole size, $`\lambda _H\begin{array}{c}>\hfill \\ \hfill \end{array}R_{bh}`$. So the black hole emits predominantly into s-wave modes with $`\mathrm{}=0`$. The black hole does not make use of the large number of highly degenerate $`\mathrm{}`$-angular modes that the higher-dimensional bulk provides. Since s-waves are non-degenerate, their number of modes is the same whether the field propagates in four or $`4+n`$ dimensions, so each brane mode is emitted at roughly the same rate as a bulk mode. In other words, the black hole does not see the angular directions, only the radial direction, which is common to modes propagating in any number of dimensions. In typical brane-world scenarios there are $`60`$ Standard Model modes propagating along the brane, versus a single graviton propagating in the bulk. Then the black hole will radiate mainly on the brane.

The evaporation time for one of these black holes will be extremely short. The black hole will typically be only a factor $`O(10)`$ larger than the TeV scale, so its lifetime will be $`\tau O(10)`$ TeV$`{}_{}{}^{1}10^{27}`$ s, much shorter-lived than a typical hadronic resonance. The collider signatures for this Hawking radiation were analyzed in . One should look for events with high multiplicity, due to the large number of quanta emitted during the evaporation. Events could be tagged by the presence of promt leptons and photons, of energy $`O(100)`$ GeV, and of similarly very energetic jets.

## 5 String balls

The semiclassical description of a black hole must break down when its mass is close to the fundamental mass scale of gravity $`M_{}`$. But in string theory, if the string coupling $`g_s`$ is weak, this description breaks down even at higher masses. When the black hole radius $`R_H`$ gets smaller than the string length $`\mathrm{}_s`$ (and the black hole temperature gets higher than the Hagedorn temperature), the string corrections to the low energy effective action cannot be argued, as we did before, to be hidden inside the horizon, so one must resort to a stringy description. The critical mass where this happens, i.e., the minimum mass for which a semiclassical description of the black hole is reliable, is

$$M_{min}M_s/g_s^2,$$
(4)
and, for small $`g_s`$, this can be quite larger than $`M_{}g_s^{2/n+2}M_s`$. In particular, this means that if $`g_s\begin{array}{c}<\hfill \\ \hfill \end{array}0.3`$, the LHC may not be able to probe the regime of black holes.

In this case, what one would be producing, instead of a black hole, is a highly excited configuration, a long, jagged, very massive string resonance: a string ball999The phrase “ball of string” had been used before in a slightly different context in .. Over the years, a concept of correspondence between black holes and strings has been developed , according to which a black hole that evaporates down to the mass (4) makes a transition into a string ball. Subsequently, the string ball evaporates by emitting massless string modes at the Hagedorn temperature.

The production cross section for a string ball can be computed using string perturbation theory, by factorizing the four-point amplitude for forward, elastic string scattering in the resonant $`s`$-channel – one of the earliest calculations in string theory. This yields

$$\sigma g_s^2s/M_s^4.$$
(5)
When viewed in the $`t`$ channel, this four-point amplitude is dominated by graviton exchange. The latter indeed determines both the factor $`g_s^2`$, from tree-level exchange of a closed-string state, and the energy dependence $`s`$, from the graviton pole. Therefore the result is universal for all string theories, since they all contain a graviton. So, besides factors that depend on the polarization of the ingoing particles, the result is independent of the initial states, as long as they are much lighter than $`\sqrt{s}`$.

This perturbative calculation will receive corrections at energies where the unitarity bounds are saturated, $`g_s^2s/M_s^21`$. Above these energies, it seems reasonable to assume that the cross section saturates and remains constant at $`\sigma l_s^2M_s^2`$. This cross section then extends up to the energies (4) where the black holes form, and interpolates between all these regimes in a parametrically smooth manner. This simple argument provides evidence in favor of the geometric cross section (3).

It is clear that in scenarios with string scale near the TeV, string balls will be produced at rates as large as the black hole production rates discussed above. In fact, it may happen that only the string ball regime is accesible to the LHC. The decay modes and properties of string balls are also quite similar to those of black holes. A main difference is that the temperature of the radiation emitted (the average energy of the decay products) is fixed at the Hagedorn temperature, and is essentially independent of the mass of the string ball, whereas for black holes the temperature is smaller for larger black holes. This may help distinguish between both kinds of objects, and may also allow for a determination of parameters such as the string mass $`M_s`$.

Acknowledgements

I would like to thank the organizers of the XIIth Japanese General Relativity and Gravitation Meeting, and in particular Tetsuya Shiromizu, for the invitation to such a delightful conference.