Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: KinyaBERT: a Morphology-aware Kinyarwanda Language Model   Pre-trained language models such as BERT have been successful at tackling
many natural language processing tasks. However, the unsupervised sub-word
tokenization methods commonly used in these models (e.g., byte-pair encoding -
BPE) are sub-optimal at handling morphologically rich languages. Even given a
morphological analyzer, naive sequencing of morphemes into a standard BERT
architecture is inefficient at capturing morphological compositionality and
expressing word-relative syntactic regularities. We address these challenges by
proposing a simple yet effective two-tier BERT architecture that leverages a
morphological analyzer and explicitly represents morphological
compositionality. Despite the success of BERT, most of its evaluations have
been conducted on high-resource languages, obscuring its applicability on
low-resource languages. We evaluate our proposed method on the low-resource
morphologically rich Kinyarwanda language, naming the proposed model
architecture KinyaBERT. A robust set of experimental results reveal that
KinyaBERT outperforms solid baselines by 2% in F1 score on a named entity
recognition task and by 4.3% in average score of a machine-translated GLUE
benchmark. KinyaBERT fine-tuning has better convergence and achieves more
robust results on multiple tasks even in the presence of translation noise.

7
The categories of this paper are: hep-th
20
The categories of this paper are: acc-phys
11
The categories of this paper are: hep-lat
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
15
The categories of this paper are: nucl-ex
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
19
The categories of this paper are: eess
26
The categories of this paper are: bayes-an
18
The categories of this paper are: econ
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
25
The categories of this paper are: patt-sol