Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Linguistically Informed Masking for Representation Learning in the
  Patent Domain   Domain-specific contextualized language models have demonstrated substantial
effectiveness gains for domain-specific downstream tasks, like similarity
matching, entity recognition or information retrieval. However successfully
applying such models in highly specific language domains requires domain
adaptation of the pre-trained models. In this paper we propose the empirically
motivated Linguistically Informed Masking (LIM) method to focus
domain-adaptative pre-training on the linguistic patterns of patents, which use
a highly technical sublanguage. We quantify the relevant differences between
patent, scientific and general-purpose language and demonstrate for two
different language models (BERT and SciBERT) that domain adaptation with LIM
leads to systematically improved representations by evaluating the performance
of the domain-adapted representations of patent language on two independent
downstream tasks, the IPC classification and similarity matching. We
demonstrate the impact of balancing the learning from different information
sources during domain adaptation for the patent domain. We make the source code
as well as the domain-adaptive pre-trained patent language models publicly
available at https://github.com/sophiaalthammer/patent-lim.

13
The categories of this paper are: quant-ph
18
The categories of this paper are: econ
24
The categories of this paper are: dg-ga
31
The categories of this paper are: ao-sci
33
The categories of this paper are: plasm-ph
3
The categories of this paper are: physics
36
The categories of this paper are: cmp-lg
2
The categories of this paper are: math
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
16
The categories of this paper are: stat
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
28
The categories of this paper are: alg-geom
35
The categories of this paper are: mtrl-th