Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: How Does Fine-tuning Affect the Geometry of Embedding Space: A Case
  Study on Isotropy   It is widely accepted that fine-tuning pre-trained language models usually
brings about performance improvements in downstream tasks. However, there are
limited studies on the reasons behind this effectiveness, particularly from the
viewpoint of structural changes in the embedding space. Trying to fill this
gap, in this paper, we analyze the extent to which the isotropy of the
embedding space changes after fine-tuning. We demonstrate that, even though
isotropy is a desirable geometrical property, fine-tuning does not necessarily
result in isotropy enhancements. Moreover, local structures in pre-trained
contextual word representations (CWRs), such as those encoding token types or
frequency, undergo a massive change during fine-tuning. Our experiments show
dramatic growth in the number of elongated directions in the embedding space,
which, in contrast to pre-trained CWRs, carry the essential linguistic
knowledge in the fine-tuned embedding space, making existing isotropy
enhancement methods ineffective.

12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
26
The categories of this paper are: bayes-an
7
The categories of this paper are: hep-th
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
27
The categories of this paper are: comp-gas