Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Gradient-based Hyperparameter Optimization Over Long Horizons   Gradient-based hyperparameter optimization has earned a widespread popularity
in the context of few-shot meta-learning, but remains broadly impractical for
tasks with long horizons (many gradient steps), due to memory scaling and
gradient degradation issues. A common workaround is to learn hyperparameters
online, but this introduces greediness which comes with a significant
performance drop. We propose forward-mode differentiation with sharing (FDS), a
simple and efficient algorithm which tackles memory scaling issues with
forward-mode differentiation, and gradient degradation issues by sharing
hyperparameters that are contiguous in time. We provide theoretical guarantees
about the noise reduction properties of our algorithm, and demonstrate its
efficiency empirically by differentiating through $\sim 10^4$ gradient steps of
unrolled optimization. We consider large hyperparameter search ranges on
CIFAR-10 where we significantly outperform greedy gradient-based alternatives,
while achieving $\times 20$ speedups compared to the state-of-the-art black-box
methods. Code is available at: \url{https://github.com/polo5/FDS}

15
The categories of this paper are: nucl-ex
28
The categories of this paper are: alg-geom
26
The categories of this paper are: bayes-an
2
The categories of this paper are: math
35
The categories of this paper are: mtrl-th
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
12
The categories of this paper are: nucl-th
13
The categories of this paper are: quant-ph
4
The categories of this paper are: cond-mat
23
The categories of this paper are: solv-int
9
The categories of this paper are: nlin
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
14
The categories of this paper are: math-ph
34
The categories of this paper are: chem-ph