  Human culture relies on innovation: our ability to continuously explore how
existing elements can be combined to create new ones. Innovation is not
solitary, it relies on collective search and accumulation. Reinforcement
learning (RL) approaches commonly assume that fully-connected groups are best
suited for innovation. However, human laboratory and field studies have shown
that hierarchical innovation is more robustly achieved by dynamic social
network structures. In dynamic settings, humans oscillate between innovating
individually or in small clusters, and then sharing outcomes with others. To
our knowledge, the role of social network structure on innovation has not been
systematically studied in RL. Here, we use a multi-level problem setting
(WordCraft), with three different innovation tasks to test the hypothesis that
the social network structure affects the performance of distributed RL
algorithms. We systematically design networks of DQNs sharing experiences from
their replay buffers in varying structures (fully-connected, small world,
dynamic, ring) and introduce a set of behavioral and mnemonic metrics that
extend the classical reward-focused evaluation framework of RL. Comparing the
level of innovation achieved by different social network structures across
different tasks shows that, first, consistent with human findings, experience
sharing within a dynamic structure achieves the highest level of innovation in
tasks with a deceptive nature and large search spaces. Second, experience
sharing is not as helpful when there is a single clear path to innovation.
Third, the metrics we propose, can help understand the success of different
social network structures on different tasks, with the diversity of experiences
on an individual and group level lending crucial insights.
