Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Provable Continual Learning via Sketched Jacobian Approximations   An important problem in machine learning is the ability to learn tasks in a
sequential manner. If trained with standard first-order methods most models
forget previously learned tasks when trained on a new task, which is often
referred to as catastrophic forgetting. A popular approach to overcome
forgetting is to regularize the loss function by penalizing models that perform
poorly on previous tasks. For example, elastic weight consolidation (EWC)
regularizes with a quadratic form involving a diagonal matrix build based on
past data. While EWC works very well for some setups, we show that, even under
otherwise ideal conditions, it can provably suffer catastrophic forgetting if
the diagonal matrix is a poor approximation of the Hessian matrix of previous
tasks. We propose a simple approach to overcome this: Regularizing training of
a new task with sketches of the Jacobian matrix of past data. This provably
enables overcoming catastrophic forgetting for linear models and for wide
neural networks, at the cost of memory. The overarching goal of this paper is
to provided insights on when regularization-based continual learning algorithms
work and under what memory costs.

19
The categories of this paper are: eess
5
The categories of this paper are: gr-qc
8
The categories of this paper are: hep-ex
28
The categories of this paper are: alg-geom
4
The categories of this paper are: cond-mat
22
The categories of this paper are: chao-dyn
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
27
The categories of this paper are: comp-gas
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
35
The categories of this paper are: mtrl-th
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph