Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Model Fusion via Optimal Transport   Combining different models is a widely used paradigm in machine learning
applications. While the most common approach is to form an ensemble of models
and average their individual predictions, this approach is often rendered
infeasible by given resource constraints in terms of memory and computation,
which grow linearly with the number of models. We present a layer-wise model
fusion algorithm for neural networks that utilizes optimal transport to (soft-)
align neurons across the models before averaging their associated parameters.
  We show that this can successfully yield "one-shot" knowledge transfer (i.e,
without requiring any retraining) between neural networks trained on
heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings , we
illustrate that our approach significantly outperforms vanilla averaging, as
well as how it can serve as an efficient replacement for the ensemble with
moderate fine-tuning, for standard convolutional networks (like VGG11),
residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10,
CIFAR100, and MNIST. Finally, our approach also provides a principled way to
combine the parameters of neural networks with different widths, and we explore
its application for model compression. The code is available at the following
link, https://github.com/sidak/otfusion.

20
The categories of this paper are: acc-phys
10
The categories of this paper are: q-bio
13
The categories of this paper are: quant-ph
4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph
3
The categories of this paper are: physics
2
The categories of this paper are: math
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
30
The categories of this paper are: q-alg
35
The categories of this paper are: mtrl-th
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph