  In an Ideal Shortest Path Algorithm (ISPA), at each moment each router in a
network sends all of its traffic down the path that will incur the lowest cost
to that traffic. In the limit of an infinitesimally small amount of traffic for
a particular router, its routing that traffic via an ISPA is optimal, as far as
cost incurred by that traffic is concerned. We demonstrate though that in many
cases, due to the side-effects of one router's actions on another routers
performance, having routers use ISPA's is suboptimal as far as global aggregate
cost is concerned, even when only used to route infinitesimally small amounts
of traffic. As a particular example of this we present an instance of Braess'
paradox for ISPA's, in which adding new links to a network decreases overall
throughput. We also demonstrate that load-balancing, in which the routing
decisions are made to optimize the global cost incurred by all traffic
currently being routed, is suboptimal as far as global cost averaged across
time is concerned. This is also due to "side-effects", in this case of current
routing decision on future traffic.
  The theory of COllective INtelligence (COIN) is concerned precisely with the
issue of avoiding such deleterious side-effects. We present key concepts from
that theory and use them to derive an idealized algorithm whose performance is
better than that of the ISPA, even in the infinitesimal limit. We present
experiments verifying this, and also showing that a machine-learning-based
version of this COIN algorithm in which costs are only imprecisely estimated (a
version potentially applicable in the real world) also outperforms the ISPA,
despite having access to less information than does the ISPA. In particular,
this COIN algorithm avoids Braess' paradox.
