Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: BERTQA -- Attention on Steroids   In this work, we extend the Bidirectional Encoder Representations from
Transformers (BERT) with an emphasis on directed coattention to obtain an
improved F1 performance on the SQUAD2.0 dataset. The Transformer architecture
on which BERT is based places hierarchical global attention on the
concatenation of the context and query. Our additions to the BERT architecture
augment this attention with a more focused context to query (C2Q) and query to
context (Q2C) attention via a set of modified Transformer encoder units. In
addition, we explore adding convolution-based feature extraction within the
coattention architecture to add localized information to self-attention. We
found that coattention significantly improves the no answer F1 by 4 points in
the base and 1 point in the large architecture. After adding skip connections
the no answer F1 improved further without causing an additional loss in has
answer F1. The addition of localized feature extraction added to attention
produced an overall dev F1 of 77.03 in the base architecture. We applied our
findings to the large BERT model which contains twice as many layers and
further used our own augmented version of the SQUAD 2.0 dataset created by back
translation, which we have named SQUAD 2.Q. Finally, we performed
hyperparameter tuning and ensembled our best models for a final F1/EM of
82.317/79.442 (Attention on Steroids, PCE Test Leaderboard).

9
The categories of this paper are: nlin
21
The categories of this paper are: adap-org
16
The categories of this paper are: stat
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
22
The categories of this paper are: chao-dyn
32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
37
The categories of this paper are: supr-con