|A | linear functions. The main differences between the methods are that the QMDP and fast informed bound methods update linear functions directly, and they guarantee upper bounds and unique convergence.

A more sophisticated parametric model of a convex function is the softmax model (Parr
& Russell, 1995):

$$\hat{V}(b)=\left[\sum_{\alpha\in\Gamma}\left[\sum_{s\in S}\alpha(s)b(s)\right]^{k}\right]^{\frac{1}{k}},$$

where Γ is the set of linear functions α with adaptive parameters to fit and k is a "temperature" parameter that provides a better fit to the underlying piecewise linear convex function for larger values. The function represents a soft approximation of a piecewise linear convex function, with the parameter k smoothing the approximation.

## 4.6.4 Control

We tested the control performance of the least-squares approach on the linear Q-function model (Littman et al., 1995) and the softmax model (Parr & Russell, 1995).  For the softmax model we varied the number of linear functions, trying cases with 10 and 15 linear functions respectively. In the first set of experiments we used parallel (synchronous) updates and samples at a fixed set of 100 belief points.  We applied stochastic gradient descent techniques to find the best fit in both cases. We tested the control performance for value-function approximations obtained after 10, 20 and 30 updates, starting from the QMDP solution. In the second set of experiments, we applied the incremental stochastic update scheme with Gauss-Seidel-style updates. The results for this method were acquired after every grid point was updated 150 times, with learning rates decreasing linearly in the range between 0.2 and 0.001. Again we started from the QMDP solution. The results for lookahead controllers are summarized in Figure 22, which also shows the control performance of the direct Q-function controller and, for comparison, the results for the QMDP method.

The linear-Q function model performed very well and the results for the lookahead design were better than the results for the QMDP method. The difference was quite apparent for direct approaches. In general, the good performance of the method can be attributed to the choice of a function model that let us match the shape of the optimal value function reasonably well. In contrast, the softmax models (with 10 and 15 linear functions) did not perform as expected. This is probably because in the softmax model all linear functions are updated for every sample point. This leads to situations in which multiple linear functions try to track a belief point during its update. Under these circumstances it is hard to capture the structure of the optimal value function accurately. The other negative feature is that the effects of on-line changes of all linear functions are added in the softmax approximation, and thus could bias incremental update schemes. In the ideal case, we would like to identify one vector α responsible for a specific belief point and update (modify) only that vector.

The linear Q-function approach avoids this problem by always updating only a single linear function (corresponding to an action).