  With the prevalence of publicly available source code repositories to train
deep neural network models, neural program models can do well in source code
analysis tasks such as predicting method names in given programs that cannot be
easily done by traditional program analysis techniques. Although such neural
program models have been tested on various existing datasets, the extent to
which they generalize to unforeseen source code is largely unknown. Since it is
very challenging to test neural program models on all unforeseen programs, in
this paper, we propose to evaluate the generalizability of neural program
models with respect to semantic-preserving transformations: a generalizable
neural program model should perform equally well on programs that are of the
same semantics but of different lexical appearances and syntactical structures.
We compare the results of various neural program models for the method name
prediction task on programs before and after automated semantic-preserving
transformations. We use three Java datasets of different sizes and three
state-of-the-art neural network models for code, namely code2vec, code2seq, and
GGNN, to build nine such neural program models for evaluation. Our results show
that even with small semantically preserving changes to the programs, these
neural program models often fail to generalize their performance. Our results
also suggest that neural program models based on data and control dependencies
in programs generalize better than neural program models based only on abstract
syntax trees. On the positive side, we observe that as the size of the training
dataset grows and diversifies the generalizability of correct predictions
produced by the neural program models can be improved too. Our results on the
generalizability of neural program models provide insights to measure their
limitations and provide a stepping stone for their improvement.
