Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better
  Language Models for Code Understanding   Code understanding is an increasingly important application of Artificial
Intelligence. A fundamental aspect of understanding code is understanding text
about code, e.g., documentation and forum discussions. Pre-trained language
models (e.g., BERT) are a popular approach for various NLP tasks, and there are
now a variety of benchmarks, such as GLUE, to help improve the development of
such models for natural language understanding. However, little is known about
how well such models work on textual artifacts about code, and we are unaware
of any systematic set of downstream tasks for such an evaluation. In this
paper, we derive a set of benchmarks (BLANCA - Benchmarks for LANguage models
on Coding Artifacts) that assess code understanding based on tasks such as
predicting the best answer to a question in a forum post, finding related forum
posts, or predicting classes related in a hierarchy from class documentation.
We evaluate the performance of current state-of-the-art language models on
these tasks and show that there is a significant improvement on each task from
fine tuning. We also show that multi-task training over BLANCA tasks helps
build better language models for code understanding.

24
The categories of this paper are: dg-ga
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
8
The categories of this paper are: hep-ex
3
The categories of this paper are: physics
16
The categories of this paper are: stat
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
29
The categories of this paper are: funct-an
20
The categories of this paper are: acc-phys
27
The categories of this paper are: comp-gas
0
The categories of this paper are: hep-ph