Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: AlgebraNets   Neural networks have historically been built layerwise from the set of
functions in ${f: \mathbb{R}^n \to \mathbb{R}^m }$, i.e. with activations and
weights/parameters represented by real numbers, $\mathbb{R}$. Our work
considers a richer set of objects for activations and weights, and undertakes a
comprehensive study of alternative algebras as number representations by
studying their performance on two challenging problems: large-scale image
classification using the ImageNet dataset and language modeling using the
enwiki8 and WikiText-103 datasets. We denote this broader class of models as
AlgebraNets. Our findings indicate that the conclusions of prior work, which
explored neural networks constructed from $\mathbb{C}$ (complex numbers) and
$\mathbb{H}$ (quaternions) on smaller datasets, do not always transfer to these
challenging settings. However, our results demonstrate that there are
alternative algebras which deliver better parameter and computational
efficiency compared with $\mathbb{R}$. We consider $\mathbb{C}$, $\mathbb{H}$,
$M_{2}(\mathbb{R})$ (the set of $2\times2$ real-valued matrices),
$M_{2}(\mathbb{C})$, $M_{3}(\mathbb{R})$ and $M_{4}(\mathbb{R})$. Additionally,
we note that multiplication in these algebras has higher compute density than
real multiplication, a useful property in situations with inherently limited
parameter reuse such as auto-regressive inference and sparse neural networks.
We therefore investigate how to induce sparsity within AlgebraNets. We hope
that our strong results on large-scale, practical benchmarks will spur further
exploration of these unconventional architectures which challenge the default
choice of using real numbers for neural network weights and activations.

24
The categories of this paper are: dg-ga
11
The categories of this paper are: hep-lat
32
The categories of this paper are: atom-ph
19
The categories of this paper are: eess
28
The categories of this paper are: alg-geom
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
6
The categories of this paper are: astro-ph
34
The categories of this paper are: chem-ph
12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
4
The categories of this paper are: cond-mat