Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting
  BERT   By introducing a small set of additional parameters, a probe learns to solve
specific linguistic tasks (e.g., dependency parsing) in a supervised manner
using feature representations (e.g., contextualized embeddings). The
effectiveness of such probing tasks is taken as evidence that the pre-trained
model encodes linguistic knowledge. However, this approach of evaluating a
language model is undermined by the uncertainty of the amount of knowledge that
is learned by the probe itself. Complementary to those works, we propose a
parameter-free probing technique for analyzing pre-trained language models
(e.g., BERT). Our method does not require direct supervision from the probing
tasks, nor do we introduce additional parameters to the probing process. Our
experiments on BERT show that syntactic trees recovered from BERT using our
method are significantly better than linguistically-uninformed baselines. We
further feed the empirically induced dependency structures into a downstream
sentiment classification task and find its improvement compatible with or even
superior to a human-designed dependency schema.

18
The categories of this paper are: econ
32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th
20
The categories of this paper are: acc-phys
29
The categories of this paper are: funct-an
34
The categories of this paper are: chem-ph
10
The categories of this paper are: q-bio
2
The categories of this paper are: math
28
The categories of this paper are: alg-geom
0
The categories of this paper are: hep-ph
35
The categories of this paper are: mtrl-th
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat