  Despite an ever growing number of word representation models introduced for a
large number of languages, there is a lack of a standardized technique to
provide insights into what is captured by these models. Such insights would
help the community to get an estimate of the downstream task performance, as
well as to design more informed neural architectures, while avoiding extensive
experimentation which requires substantial computational resources not all
researchers have access to. A recent development in NLP is to use simple
classification tasks, also called probing tasks, that test for a single
linguistic feature such as part-of-speech. Existing studies mostly focus on
exploring the linguistic information encoded by the continuous representations
of English text. However, from a typological perspective the morphologically
poor English is rather an outlier: the information encoded by the word order
and function words in English is often stored on a morphological level in other
languages. To address this, we introduce 15 type-level probing tasks such as
case marking, possession, word length, morphological tag count and pseudoword
identification for 24 languages. We present a reusable methodology for creation
and evaluation of such tests in a multilingual setting. We then present
experiments on several diverse multilingual word embedding models, in which we
relate the probing task performance for a diverse set of languages to a range
of five classic NLP tasks: POS-tagging, dependency parsing, semantic role
labeling, named entity recognition and natural language inference. We find that
a number of probing tests have significantly high positive correlation to the
downstream tasks, especially for morphologically rich languages. We show that
our tests can be used to explore word embeddings or black-box neural models for
linguistic cues in a multilingual setting.
