Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Representation Collapse of Sparse Mixture of Experts   Sparse mixture of experts provides larger model capacity while requiring a
constant computational overhead. It employs the routing mechanism to distribute
input tokens to the best-matched experts according to their hidden
representations. However, learning such a routing mechanism encourages token
clustering around expert centroids, implying a trend toward representation
collapse. In this work, we propose to estimate the routing scores between
tokens and experts on a low-dimensional hypersphere. We conduct extensive
experiments on cross-lingual language model pre-training and fine-tuning on
downstream tasks. Experimental results across seven multilingual benchmarks
show that our method achieves consistent gains. We also present a comprehensive
analysis on the representation and routing behaviors of our models. Our method
alleviates the representation collapse issue and achieves more consistent
routing than the baseline mixture-of-experts methods.

22
The categories of this paper are: chao-dyn
33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
6
The categories of this paper are: astro-ph
4
The categories of this paper are: cond-mat
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
20
The categories of this paper are: acc-phys
15
The categories of this paper are: nucl-ex
25
The categories of this paper are: patt-sol
18
The categories of this paper are: econ
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph
12
The categories of this paper are: nucl-th
2
The categories of this paper are: math