it should delay how quickly the learners determine their optimal policies, but it won't affect the quality (for G ) of those policies finally arrived at. Note also that trying to determine whether some particular xr n , r ′′′ ( t E κ ( τ )) should be included in xr , a ( τ )'s effect set would mean, in part, determining whether packets routed from ( r, d ) would have reached r ′′ if
( r, d ) had made some routing decision different from the one it actually made. This would be a non-trivial exercise, in general.

In contrast to the case with the x r n , r n ( t ), there are X r n , r ( t ) with t in the future of τ 's wave that both are affected by x r d ( t ) and also are not excluded by any of our approximations so far. In particular, the X,",a(t) with either r" = r or r" one hop away from r will be directly affected by xr,a(t), for t E U,V,,,,,(t +iL)) (cf. the definition of the X variables).

For simplicity, we restrict consideration of such X ro d variables to those with the same router as r, r" = r, This final estimate for the effect set is clearly rather poor - presumably results better than those presented below would accrue to use of a more accurate effect set. However it's worth bearing in mind that there is a "self-stabilizing" nature to the choice of effect sets, when used in conjunction with effect set WLU's. This nature is mediated by the learning algorithms. If one assigns the same utility function to two agents, then the reward one agent gets will be determined in part by what the other one does.  So as it modifies its behavior to try to increase its reward, that first agent will be modifying its behavior in a way dependent on what the other agent does. In other words, if two agents are given the same WLU because they are estimated to be in each other's effect set, then ipso facto they will be in each other's effect set Using our estimate for the effect set, the WLU for ( η, τ ) is given by the difference between the total cost accrued in  's wave by all agents in the network and the cost accrued by agents when all agents sharing η's destination are "erased."  More precisely, any agent η that has a destination d will have the following effect set WLU's, gn,r:

9n,r(c)=  G(c)= G(CL r,r,r(c)) E x, xr d (t) V,, l Xr xr',d'(t) Vr'( EXr',d''(t)) - EXr',d'(t) Vr'( Exr r , d ( t ) [ Vr + ( E X r + , a + ( t )) - Vr + ( E X r + , a + ( t ))] (3) where I(.) is the indicator function that equals 1 if its argument is true, 0 otherwise.
To allow the learner to receive feedback concerning its actions in a wave immediately following that wave rather than wait for ~ WL time steps, we will approximate the second sum in that last equality, the one over times following τ's wave, as zero. There is another way we can view the resultant expression, rather than as an approximation to the effect