V (), to re e t the fa t that real networks have dieren es in router software and hardware

(response time, queue length, pro essing speed et
). For simpli ity, W is the same for all routers however. With these denitions, world utility is given by

$$G(\underline{{{\zeta}}})$$
G( ) = X
$$=\sum_{t,r}\,z_{r}(t)\,\,\,V_{r}(\,Z_{r}(t))\tag{2}$$ $$=\sum_{t,r,d}x_{r,d}(t)V_{r}\left(Z_{r}(t)\right)$$ $$=\sum_{t,r,d}x_{r,d}(t)V_{r}\left(\frac{1}{W}\sum_{t^{\prime}=t-W+1}^{t}\sum_{d^{\prime}}x_{r,d^{\prime}}\left(t^{\prime}\right)\right)$$ $$=\sum_{t,r,d}x_{r,d}(t)V_{r}\left(\sum_{d^{\prime}}X_{r,d^{\prime}}\left(t\right)\right)\,\,\,.$$

Our equation for G expli itly demonstrates that, as laimed above, in our representation we an express G( ) as a sum of rewards, Pt Rt(
;t), where R(;t) 
an be written as fun
tion of a pair of (r; d)-indexed ve tors:

$$R_{t}\big(x_{r,d}(t),X_{r,d}(t)\big)=\sum_{r,d}x_{r,d}(t)V_{r}\left(\sum_{d^{\prime}}X_{r,d^{\prime}}\left(t\right)\right)\,.$$

Also as laimed, the Rt are temporal translations of one another.

Given this model, some of the omponents of 

;t must be identied with the values xr;d;r0 (t) 8 r; d; r0and t, sin
e those x's are set by the a
tions the agents will take. Sin
e all arguments of G must be omponents of  , we also in lude the Xr;d(t) 8r; d; t as 
omponents of 

;t . Formally, for routing based on ML agents, the internal parameters of the ML agents must also be in luded in  . This is be ause those parameters ae t the routing, and in turn are ae ted by it. So to have  evolve deterministi ally, sin e it in ludes the routing variables, it must also ontain internal parameters of the agents. We won't have any need to expli itly delineate su h variables here however, and will mostly phrase the dis ussion as though there were no su h internal parameters.

Now the values fxr;d;r0 (t  1)g 8r; d; r0spe
ify the values fxr;d(t)g 8r; d dire
tly. Therefore, in on ert with the fxr;d(t 0 < t)g, they also set the fXr;d(t)g dire
tly. Moreover in our simulations the de isions fxr;d;r0 (t)g 8r; d; r0xed by the routing algorithms at all times t are given by a xed fun tion of the fxr;d(t)g and the fZr (t) = Pd0 Xr;d0 (t)g. So in point of fa t we an map the set of fxr;d;r0 (t  1); Xr;d0 (t)g 8r; d; r0to the full set fxr;d;r0 (t)g 8r; d; r0, not just to fxr;d(t)g. A

ordingly, the xr;d;r0 undergo deterministi evolution. Sin
e their values a ross time set all the values of the Xr;d(t) a
ross time, we see that the entire set of the omponents of 

;t undergo deterministi evolution in this representation, as required.

For evaluating the wonderful life utility we will need to group the omponents of 

;t into disjoint agents . Here we will have two types of agent, both types being indexed by router-destination pairs. For ea h su h agent index (r; d), the rst agent type is the variable Xr;d(t), and the se
ond agent type is the Eu
lidean ve
tor with 
omponents indexed by r 0

,

(xr;d)r0 (t). In setting \a
tions" we are 
on
erned with setting the states of the agents of the se ond type. A

ordingly, our learners will all be asso iated with agents of this se ond
