  In both the human brain and any general artificial intelligence (AI), a
representation of the past is necessary to predict the future. However, perfect
storage of all experiences is not feasible. One approach utilized in many
applications, including reward prediction in reinforcement learning, is to
retain recently active features of experience in a buffer. Despite its prior
successes, we show that the fixed length buffer renders Deep Q-learning
Networks (DQNs) fragile to changes in the scale over which information can be
learned. To enable learning when the relevant temporal scales in the
environment are not known *a priori*, recent advances in psychology and
neuroscience suggest that the brain maintains a compressed representation of
the past. Here we introduce a neurally-plausible, scale-free memory
representation we call Scale-Invariant Temporal History (SITH) for use with
artificial agents. This representation covers an exponentially large period of
time by sacrificing temporal accuracy for events further in the past. We
demonstrate the utility of this representation by comparing the performance of
agents given SITH, buffer, and exponential decay representations in learning to
play video games at different levels of complexity. In these environments, SITH
exhibits better learning performance by storing information for longer
timescales than a fixed-size buffer, and representing this information more
clearly than a set of exponentially decayed features. Finally, we discuss how
the application of SITH, along with other human-inspired models of cognition,
could improve reinforcement and machine learning algorithms in general.
