  The representational drift observed in neural populations raises serious
questions about how accurate decoding survives these changes. In primary visual
cortex, it is hotly debated whether such variation is a direct tuning shift
that would corrupt decoding or if it can be explained by changes in behavioral
or internal state, which could be compensated by joint encoding of the stimulus
and the state. We estimate the effects of stimulus-relevant representational
drift on decoding using a publicly accessible dataset of mouse V1 responses to
a natural movie. Because the only invariant component of the sensory experience
across all 24 animals is that they all watch the same natural movie, we can
learn a subject-invariant efficient neural representation that retains only
stimulus-relevant components. We use contrastive learning between the neural
response and the stimulus to learn a neural representation for
stimulus-relevant features. This learned representation minimizes decoding
error as quantified by Bayes risk. We show that it can be used to read out
behaviorally relevant stimulus features (time, static scene, optic flow, and
joint spatio-temporal features) at 33ms resolution accurately, a finer
timescale than what has previously been explored. When we use the model trained
on one recording session to derive feature activations on another, decoding
performance is reduced by approximately $40\%$. Motion encoding is most
susceptible to representational drift. In addition, when we enlarge the error
tolerance window to 1sec, we recover stable stimulus encoding across sessions,
echoing previous findings. This shows that decoding stimulus features that vary
on fast timescales may require complex computation downstream of V1 to
compensate for representational drift.
