  Analysis of pre-trained code models also has revealed that they can
effectively learn program syntax. However, these works are limited in analyzing
code syntax and their distance-based approaches are not accurate due to the
curse of high dimensionality. Furthermore, the study of the learnt program
semantics of these models is rarely discussed. To further understand the code
features learnt by these models, in this paper, we target two well-known
representative code pre-trained models (i.e., CodeBERT and GraphCodeBERT) and
devise a set of probing tasks for the syntax and semantics analysis.
Specifically, on one hand, we design two probing tasks (i.e., syntax pair node
prediction and token tagging prediction) to manipulate AST for the
understanding of learnt program syntax. On the other hand, we design two tasks
(i.e., semantic relationship prediction and semantic propagation
prediction(inGraph) ) on the constructed control flow graph (CFG), data
dependency graph (DDG) and control dependency graph (CDG) for the learnt
program semantic analysis. In addition, to understand which kind of program
semantics these pre-trained models can comprehend well, we conduct the
statistical analysis for attention weights learnt by different heads and
layers. Through extensive analysis in terms of program syntax and semantics, we
have the following findings: 1) Both CodeBERT and GraphCodeBERT can learn the
program syntax well. 2) Both CodeBERT and GraphCodeBERT can learn program
semantics to different extents. GraphCodeBERT is superior to CodeBERT in
learning program control flow and data dependency information but has a similar
capability to CodeBERT in learning control dependency information. 3) Both
CodeBERT and GraphCodeBERT can capture program semantics in the final layer of
representation, but different attention heads and layers exhibit different
roles in learning program semantics.
