type. Unless explicitly indicated otherwise, from now on we will implicitly have that second type of agent in mind whenever we refer to a "agent" or use the symbol η .

## 4.2 Ispa Routing And Coin Routing

Based on the COIN formalism presented in Section 3 and the model described above, we now present the ISPA and COIN-based routing algorithms. At time step t, ISPA has access to all the windowed loads at time step t - 1 (i.e., it has access to Zr(t - 1) Vr), and assumes that those values will remain the same at all times > t. Note that for large window sizes and times close to t, this assumption is arbitrarily accurate. Using this assumption, in ISPA, each router sends packets along the path that it calculates will minimize the costs accumulated by its packets.

The COIN-based routing algorithms, in contrast, do not have such direct access to the Zr. So to evaluate the WLU for a agent (r,d) at any time r, such an algorithm must estimate the (primary members of the) associated effect set. This means determining what components of ζ  will, under the dynamics of the system, be changed by altering any of the components of the vector xr.d(T).

As a first approximation, we will ignore effects on traffic that anging xr.a,r ( τ ) may have that are "mediated" by the learning algorithms running in the system. That is, we ignore changes that arise due to the the effects that changing xr,a,r'(r) has on rewards, changes which induce changes in future training sets, which then in turn get mapped to changes in the { xr,d,r ( t )} (and therefore the { Xr,d ( t )}) via the learning algorithms running on the agents.

As another approximation, we will ignore effects mediated by the routing algorithms' observations of the state of the network. That is, we ignore changes in the { x r v , v , v ( t )} that varying xr d ( τ ) may cause due to associated changes in the state of the network perceived by ( r ′′, d ′ )'s routing algorithm, changes that in turn cause that algorithm to modify its routing decisions accordingly. We only consider the behavior of those routing algorithms that are (potentially) directly affected by x, a(r) in that they (potentially) have to route packets that, at time r, passed through r on the way to d. So in particular we ignore effects of xr,d(T) on the {xr" ,d' ≠ d,r"' (t)}.

Since all packets routed in a wave arrive at their destinations by the end of the wave, these approximations mean that the only xrn d , 
effect set have t in the same wave as τ . These are the only ones that are, potentially, directly affected by the { xr.d,r ( t )} by "chaining together" the sequence of xr" , d",r""( t ) that get the packets in xr,d(t) to their ultimate destination. Due to the wave nature of our simulations though, the only xr",a",r""(t) within r's wave that are affected by xr,a(t) all have d" = d.

For reasons of coding simplicity, we do not concern ourselves with whether t < τ within a given wave and then exclude some x r n d n r m ( t ) accordingly. In other words, all t within τ 's wave are treated equally.

So one set of members of xr,d(r)'s effect set is {xr",d,r""(t) Vr",d,r"",t E k(t)}. Note that some of these members will be relatively unaffected by xr,a(r) (e.g., those with r" far in the net away from r ). Again for simplicity, we do not try to determine these and exclude them. As with keeping the xrn , r , r , r , ( t < τ ), this inclusion of extra agents in our estimate of the effect set should hurt learnability, but in general should not hurt factoredness. Therefore