Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Hierarchical Representation in Neural Language Models: Suppression and
  Recovery of Expectations   Deep learning sequence models have led to a marked increase in performance
for a range of Natural Language Processing tasks, but it remains an open
question whether they are able to induce proper hierarchical generalizations
for representing natural language from linear input alone. Work using
artificial languages as training input has shown that LSTMs are capable of
inducing the stack-like data structures required to represent context-free and
certain mildly context-sensitive languages---formal language classes which
correspond in theory to the hierarchical structures of natural language. Here
we present a suite of experiments probing whether neural language models
trained on linguistic data induce these stack-like data structures and deploy
them while incrementally predicting words. We study two natural language
phenomena: center embedding sentences and syntactic island constraints on the
filler--gap dependency. In order to properly predict words in these structures,
a model must be able to temporarily suppress certain expectations and then
recover those expectations later, essentially pushing and popping these
expectations on a stack. Our results provide evidence that models can
successfully suppress and recover expectations in many cases, but do not fully
recover their previous grammatical state.

7
The categories of this paper are: hep-th
15
The categories of this paper are: nucl-ex
19
The categories of this paper are: eess
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
6
The categories of this paper are: astro-ph
23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
35
The categories of this paper are: mtrl-th
10
The categories of this paper are: q-bio
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg