Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Finding patterns in Knowledge Attribution for Transformers   We analyze the Knowledge Neurons framework for the attribution of factual and
relational knowledge to particular neurons in the transformer network. We use a
12-layer multi-lingual BERT model for our experiments. Our study reveals
various interesting phenomena. We observe that mostly factual knowledge can be
attributed to middle and higher layers of the network($\ge 6$). Further
analysis reveals that the middle layers($6-9$) are mostly responsible for
relational information, which is further refined into actual factual knowledge
or the "correct answer" in the last few layers($10-12$). Our experiments also
show that the model handles prompts in different languages, but representing
the same fact, similarly, providing further evidence for effectiveness of
multi-lingual pre-training. Applying the attribution scheme for grammatical
knowledge, we find that grammatical knowledge is far more dispersed among the
neurons than factual knowledge.

12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
11
The categories of this paper are: hep-lat
36
The categories of this paper are: cmp-lg
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas
30
The categories of this paper are: q-alg
17
The categories of this paper are: q-fin
35
The categories of this paper are: mtrl-th
37
The categories of this paper are: supr-con
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
16
The categories of this paper are: stat
24
The categories of this paper are: dg-ga