  A central challenge in the computational modeling and simulation of a
multitude of science applications is to achieve robust and accurate closures
for their coarse-grained representations due to underlying highly nonlinear
multiscale interactions. These closure models are common in many nonlinear
spatiotemporal systems to account for losses due to reduced order
representations, including many transport phenomena in fluids. Previous
data-driven closure modeling efforts have mostly focused on supervised learning
approaches using high fidelity simulation data. On the other hand,
reinforcement learning (RL) is a powerful yet relatively uncharted method in
spatiotemporally extended systems. In this study, we put forth a modular
dynamic closure modeling and discovery framework to stabilize the Galerkin
projection based reduced order models that may arise in many nonlinear
spatiotemporal dynamical systems with quadratic nonlinearity. However, a key
element in creating a robust RL agent is to introduce a feasible reward
function, which can be constituted of any difference metrics between the RL
model and high fidelity simulation data. First, we introduce a multi-modal RL
(MMRL) to discover mode-dependant closure policies that utilize the high
fidelity data in rewarding our RL agent. We then formulate a variational
multiscale RL (VMRL) approach to discover closure models without requiring
access to the high fidelity data in designing the reward function.
Specifically, our chief innovation is to leverage variational multiscale
formalism to quantify the difference between modal interactions in Galerkin
systems. Our results in simulating the viscous Burgers equation indicate that
the proposed VMRL method leads to robust and accurate closure
parameterizations, and it may potentially be used to discover scale-aware
closure models for complex dynamical systems.
