Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning Music Helps You Read: Using Transfer to Study Linguistic
  Structure in Language Models   We propose transfer learning as a method for analyzing the encoding of
grammatical structure in neural language models. We train LSTMs on
non-linguistic data and evaluate their performance on natural language to
assess which kinds of data induce generalizable structural features that LSTMs
can use for natural language. We find that training on non-linguistic data with
latent structure (MIDI music or Java code) improves test performance on natural
language, despite no overlap in surface form or vocabulary. To pinpoint the
kinds of abstract structure that models may be encoding to lead to this
improvement, we run similar experiments with two artificial parentheses
languages: one which has a hierarchical recursive structure, and a control
which has paired tokens but no recursion. Surprisingly, training a model on
either of these artificial languages leads to the same substantial gains when
testing on natural language. Further experiments on transfer between natural
languages controlling for vocabulary overlap show that zero-shot performance on
a test language is highly correlated with typological syntactic similarity to
the training language, suggesting that representations induced by pre-training
correspond to the cross-linguistic syntactic properties. Our results provide
insights into the ways that neural models represent abstract syntactic
structure, and also about the kind of structural inductive biases which allow
for natural language acquisition.

24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
17
The categories of this paper are: q-fin
4
The categories of this paper are: cond-mat
31
The categories of this paper are: ao-sci
35
The categories of this paper are: mtrl-th
21
The categories of this paper are: adap-org
0
The categories of this paper are: hep-ph
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas