  Transformer-based language models have recently achieved remarkable results
in many natural language tasks. However, performance on leaderboards is
generally achieved by leveraging massive amounts of training data, and rarely
by encoding explicit linguistic knowledge into neural models. This has led many
to question the relevance of linguistics for modern natural language
processing. In this dissertation, I present several case studies to illustrate
how theoretical linguistics and neural language models are still relevant to
each other. First, language models are useful to linguists by providing an
objective tool to measure semantic distance, which is difficult to do using
traditional methods. On the other hand, linguistic theory contributes to
language modelling research by providing frameworks and sources of data to
probe our language models for specific aspects of language understanding.
  This thesis contributes three studies that explore different aspects of the
syntax-semantics interface in language models. In the first part of my thesis,
I apply language models to the problem of word class flexibility. Using mBERT
as a source of semantic distance measurements, I present evidence in favour of
analyzing word class flexibility as a directional process. In the second part
of my thesis, I propose a method to measure surprisal at intermediate layers of
language models. My experiments show that sentences containing morphosyntactic
anomalies trigger surprisals earlier in language models than semantic and
commonsense anomalies. Finally, in the third part of my thesis, I adapt several
psycholinguistic studies to show that language models contain knowledge of
argument structure constructions. In summary, my thesis develops new
connections between natural language processing, linguistic theory, and
psycholinguistics to provide fresh perspectives for the interpretation of
language models.
