Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Probing as Quantifying Inductive Bias   Pre-trained contextual representations have led to dramatic performance
improvements on a range of downstream tasks. Such performance improvements have
motivated researchers to quantify and understand the linguistic information
encoded in these representations. In general, researchers quantify the amount
of linguistic information through probing, an endeavor which consists of
training a supervised model to predict a linguistic property directly from the
contextual representations. Unfortunately, this definition of probing has been
subject to extensive criticism in the literature, and has been observed to lead
to paradoxical and counter-intuitive results. In the theoretical portion of
this paper, we take the position that the goal of probing ought to be measuring
the amount of inductive bias that the representations encode on a specific
task. We further describe a Bayesian framework that operationalizes this goal
and allows us to quantify the representations' inductive bias. In the empirical
portion of the paper, we apply our framework to a variety of NLP tasks. Our
results suggest that our proposed framework alleviates many previous problems
found in probing. Moreover, we are able to offer concrete evidence that -- for
some tasks -- fastText can offer a better inductive bias than BERT.

33
The categories of this paper are: plasm-ph
29
The categories of this paper are: funct-an
7
The categories of this paper are: hep-th
19
The categories of this paper are: eess
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
34
The categories of this paper are: chem-ph
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
17
The categories of this paper are: q-fin
35
The categories of this paper are: mtrl-th