Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Geodesic Distances and Contextual Embedding Compression for Text
  Classification   In some memory-constrained settings like IoT devices and over-the-network
data pipelines, it can be advantageous to have smaller contextual embeddings.
We investigate the efficacy of projecting contextual embedding data (BERT) onto
a manifold, and using nonlinear dimensionality reduction techniques to compress
these embeddings. In particular, we propose a novel post-processing approach,
applying a combination of Isomap and PCA. We find that the geodesic distance
estimations, estimates of the shortest path on a Riemannian manifold, from
Isomap's k-Nearest Neighbors graph bolstered the performance of the compressed
embeddings to be comparable to the original BERT embeddings. On one dataset, we
find that despite a 12-fold dimensionality reduction, the compressed embeddings
performed within 0.1% of the original BERT embeddings on a downstream
classification task. In addition, we find that this approach works particularly
well on tasks reliant on syntactic data, when compared with linear
dimensionality reduction. These results show promise for a novel geometric
approach to achieve lower dimensional text embeddings from existing
transformers and pave the way for data-specific and application-specific
embedding compressions.

2
The categories of this paper are: math
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
16
The categories of this paper are: stat
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
33
The categories of this paper are: plasm-ph
12
The categories of this paper are: nucl-th
20
The categories of this paper are: acc-phys
4
The categories of this paper are: cond-mat
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph
24
The categories of this paper are: dg-ga
15
The categories of this paper are: nucl-ex
30
The categories of this paper are: q-alg
0
The categories of this paper are: hep-ph