Here, V 
(I ) denotes the optimal value fun tion maximizing E(P1 t=0 
trt) for state I . (I ; a)
is the expe ted one-step reward and equals

$$\rho(I,a\,)=\sum_{s\,\in\,S}\rho(\,s,a\,)P(\,s|I\,)=\sum_{s\,\in\,S}\sum_{s^{\prime}\,\in\,S}R(\,s,a\,,\,s^{\prime}\,)P(\,s^{\prime}|\,s,a\,)P(\,s|I\,).$$

$$\left(\,2\,\right)$$
$$\left({\mathrm{3}}\right)$$

(s; a) denotes an expe
ted one-step reward for state s and a
tion a.

Sin e the next information state I

0 =  (I ; o; a) is a deterministi fun
tion of the previous information state I , a tion a, and the observation o, the Equation 1 an be rewritten more ompa tly by summing over all possible observations :

$$V^{*}(I\,)=\operatorname*{max}_{a\,\in\,\bar{A}}\left\{\sum_{s\,\in\,S}\rho(\,s,a\,)P(s|I\,)+\gamma\sum_{o\,\in\,\Theta}P(o|I\,,a\,)V^{*}(\tau\,(I\,,o\,,a))\right\}.$$

The optimal poli y (
ontrol fun tion) : I ! A sele
ts the value-maximizing a
tion

$$\mu^{*}(I)=\arg\operatorname*{max}_{a\in A}\left\{\sum_{s\in S}\rho(s,a)P(s|I)+\gamma\sum_{o\in\Theta}P(o|I,a)V^{*}(\tau(I,o,a))\right\}\,.$$

$$\left({4}\right)$$

The value and ontrol fun tions an be also expressed in terms of a tion-value fun tions

(Q-fun tions)

$$\begin{array}{l l}{{}}&{{V^{*}(I\,)=\operatorname*{max}_{a\,\in\,A}Q^{*}(I,a)}}\\ {{}}&{{\mu^{*}(I\,)=\operatorname*{arg\,max}_{a\,\in\,A}Q^{*}(I,a),}}\\ {{}}&{{Q^{*}(I,a)=\sum_{s\,\in\,S}\rho(s,a)P(s|I\,)+\gamma\sum_{o\,\in\,\Theta}P(o|I,a)V^{*}(\tau(I,o,a\,)).}}\end{array}$$

A Q-fun tion orresponds to the expe ted reward for hosing a xed a tion (a) in the rst step and a ting optimally afterwards.

## 2.3.1 Suffi Ient Statisti S

To derive Equations 1|3 we impli itly used omplete information states. However, as remarked earlier, the information available to the de ision-maker an be also summarized by other quantities. We all them suÆ
ient information states. Su h states must preserve the ne essary information ontent and also the Markov property of the information-state de ision pro ess.

Denition 2 (SuÆ
ient information state pro ess). Let I be an information state spa e and  : I - A - - ! I be an update fun tion dening an information pro ess It =
 (It1; at1; ot ). The pro
ess is suÆ
ient with regard to the optimal 
ontrol when, for any time step t, it satises

$$\begin{array}{c}{{P(\,s_{t}|I_{t}\,)=\,P\bigl(\,s_{t}|I_{t}^{\,C}\bigr)}}\\ {{{}}}\\ {{P\bigl(\,o_{t}|I_{t-1},\,a_{t-1}\,\bigr)=\,P\bigl(\,o_{t}|I_{t-1}^{\,C},\,a_{t-1}\,\bigr),}}\end{array}$$

where IC
t and IC
t1 are 
omplete information states.

It is easy to see that Equations 1 | 3 for omplete information states must hold also for suÆ
ient information states. The key benet of suÆ
ient statisti s is that they are often
