  Program representation learning is a fundamental task in software engineering
applications. With the availability of "big code" and the development of deep
learning techniques, various program representation learning models have been
proposed to understand the semantic properties of programs and applied on
different software engineering tasks. However, no previous study has
comprehensively assessed the generalizability of these deep models on different
tasks, so that the pros and cons of the models are unclear. In this experience
paper, we try to bridge this gap by systemically evaluating the performance of
eight program representation learning models on three common tasks, where six
models are based on abstract syntax trees and two models are based on plain
text of source code. We kindly explain the criteria for selecting the models
and tasks, as well as the method for enabling end-to-end learning in each task.
The results of performance evaluation show that they perform diversely in each
task and the performance of the AST-based models is generally unstable over
different tasks. In order to further explain the results, we apply a prediction
attribution technique to find what elements are captured by the models and
responsible for the predictions in each task. Based on the findings, we discuss
some general principles for better capturing the information in the source
code, and hope to inspire researchers to improve program representation
learning methods for software engineering tasks.
