  One of the fundamental steps toward understanding a complex system is
identifying variation at the scale of the system's components that is most
relevant to behavior on a macroscopic scale. Mutual information is a natural
means of linking variation across scales of a system due to its independence of
the particular functional relationship between variables. However, estimating
mutual information given high-dimensional, continuous-valued data is
notoriously difficult, and the desideratum -- to reveal important variation in
a comprehensible manner -- is only readily achieved through exhaustive search.
Here we propose a practical, efficient, and broadly applicable methodology to
decompose the information contained in a set of measurements by lossily
compressing each measurement with machine learning. Guided by the distributed
information bottleneck as a learning objective, the information decomposition
sorts variation in the measurements of the system state by relevance to
specified macroscale behavior, revealing the most important subsets of
measurements for different amounts of predictive information. Additional
granularity is achieved by inspection of the learned compression schemes: the
variation transmitted during compression is composed of distinctions among
measurement values that are most relevant to the macroscale behavior. We focus
our analysis on two paradigmatic complex systems: a Boolean circuit and an
amorphous material undergoing plastic deformation. In both examples, specific
bits of entropy are identified out of the high entropy of the system state as
most related to macroscale behavior for insight about the connection between
micro- and macro- in the complex system. The identification of meaningful
variation in data, with the full generality brought by information theory, is
made practical for the study of complex systems.
