  A major problem in motor control is understanding how the brain plans and
executes proper movements in the face of delayed and noisy stimuli. A prominent
framework for addressing such control problems is Optimal Feedback Control
(OFC). OFC generates control actions that optimize behaviorally relevant
criteria by integrating noisy sensory stimuli and the predictions of an
internal model using the Kalman filter or its extensions. However, a
satisfactory neural model of Kalman filtering and control is lacking because
existing proposals have the following limitations: not considering the delay of
sensory feedback, training in alternating phases, and requiring knowledge of
the noise covariance matrices, as well as that of systems dynamics. Moreover,
the majority of these studies considered Kalman filtering in isolation, and not
jointly with control. To address these shortcomings, we introduce a novel
online algorithm which combines adaptive Kalman filtering with a model free
control approach (i.e., policy gradient algorithm). We implement this algorithm
in a biologically plausible neural network with local synaptic plasticity
rules. This network performs system identification and Kalman filtering,
without the need for multiple phases with distinct update rules or the
knowledge of the noise covariances. It can perform state estimation with
delayed sensory feedback, with the help of an internal model. It learns the
control policy without requiring any knowledge of the dynamics, thus avoiding
the need for weight transport. In this way, our implementation of OFC solves
the credit assignment problem needed to produce the appropriate sensory-motor
control in the presence of stimulus delay.
