Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Empirical Evaluation of Pre-trained Transformers for Human-Level NLP:
  The Role of Sample Size and Dimensionality   In human-level NLP tasks, such as predicting mental health, personality, or
demographics, the number of observations is often smaller than the standard
768+ hidden state sizes of each layer within modern transformer-based language
models, limiting the ability to effectively leverage transformers. Here, we
provide a systematic study on the role of dimension reduction methods
(principal components analysis, factorization techniques, or multi-layer
auto-encoders) as well as the dimensionality of embedding vectors and sample
sizes as a function of predictive performance. We first find that fine-tuning
large models with a limited amount of data pose a significant difficulty which
can be overcome with a pre-trained dimension reduction regime. RoBERTa
consistently achieves top performance in human-level tasks, with PCA giving
benefit over other reduction methods in better handling users that write longer
texts. Finally, we observe that a majority of the tasks achieve results
comparable to the best performance with just $\frac{1}{12}$ of the embedding
dimensions.

36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
35
The categories of this paper are: mtrl-th
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
15
The categories of this paper are: nucl-ex
31
The categories of this paper are: ao-sci
4
The categories of this paper are: cond-mat
27
The categories of this paper are: comp-gas
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
28
The categories of this paper are: alg-geom