
$$\Gamma_{i}(S)\equiv\operatorname{E}_{z|S,C_{i}}\left\{g(z)|S,C_{i}\right\}\right\}=\int_{Z}g(z)\,p(z|C_{i})\,p(S|z)\,d z\,/\,p(S|C_{i})$$

then the parameters of p(z) can be finally written 5as an unconditional expectation over the observable p(S) in a form similar to eq. (22):

$$\operatorname{E}_{z|C_{i}}\left\{g(z)|C_{i}\right\}=\operatorname{E}_{S}\left\{\Gamma_{i}(S)\operatorname{P}\{C_{i}|S\}\right\}/\operatorname{P}_{i}$$
ig C = Γ C P (25)
This expression justifies an extended form of the EM algorithm to iteratively update the parameters of p(z) by averaging Γi Ci
(S) P{ |S} over the available training information { S
( k )}
drawn from p(S). This can be considered as a numerical/statistical method for solving p(z) in the integral equation:

$$(25)$$
$$\bigu|_{z}\,p(S)$$

## P P D P Z (S|Z) (Z) Z = (S) ∫

Note that we cannot approximate p(S) as a fixed mixture in terms of p Ci
(S| ) and then computing back the corresponding p Ci
(z| ) because, in general, p(S|z) will be different for the different training examples. For the same reason, elementary deconvolution methods are not directly applicable.

This kind of problem is addressed by Vapnik (1982, 1995) to perform inference from the result of "indirect measurements". This is an ill-posed problem, requiring regularization techniques. The proposed extended EM algorithm can be considered as a method for empirical regularization, in which the solution is restricted to be in the family of mixtures of (generalized) gaussians. EM is also proposed by You and Kaveh (1996) for regularization in the context of image restoration.

The interpretation of (25) is straightforward. Since we do not know the exact z required to approximate the parameters of p(z) by empirically averaging g Ci
(z) P{ |z} , we obtain the same result by averaging the corresponding Γi Ci
(S) P{ |S} in the S domain, where Γi
(S) plays the role of g(z) in (22). As z is uncertain, g(z) is replaced by its expected value in each component given the information about S. In particular, if there is exact knowledge about the training set attributes ( S
( k )= z
( k ), i.e., R = 1 and the marginal likelihoods are impulses) then (25) reduces to
(22). Fig. 16. illustrates the approximation process performed by the extended version of the EM algorithm in a simple univariate situation.

It is convenient to develop a version of the proposed Extended EM algorithm for uncertain training sets, structured as tables of (sub)cases × (uncertainly valued) variables (see Fig. 17).

First, let us write eq. (24) expanding S in terms of its components sr:

$$p(z|C_{i},S)\,\mathrm{P}\{C_{i}|S\}=p(z,C_{i}|S)$$
$=\sum_{r}p(z,C_{i}|s_{r})\ \mathrm{P}\{s_{r}|S\}=\sum_{r}p(z|C_{i},s_{r})\ \mathrm{P}\{C_{i}|s_{r}\}\ \mathrm{P}\{s_{r}|S\}$
Therefore

$$\Gamma_{i}(S)\;\mathrm{P}\{C_{i}|S\}=\sum_{r}\;\Gamma_{i,r}(s_{r})\;\mathrm{P}\{C_{i},s_{r}|S\}$$
