  Self-supervised pre-training is gaining increasingly more popularity in
AI-aided drug discovery, leading to more and more pre-trained models with the
promise that they can extract better feature representations for molecules.
Yet, the quality of learned representations have not been fully explored. In
this work, inspired by the two phenomena of Activity Cliffs (ACs) and Scaffold
Hopping (SH) in traditional Quantitative Structure-Activity Relationship (QSAR)
analysis, we propose a method named Representation-Property Relationship
Analysis (RePRA) to evaluate the quality of the representations extracted by
the pre-trained model and visualize the relationship between the
representations and properties. The concepts of ACs and SH are generalized from
the structure-activity context to the representation-property context, and the
underlying principles of RePRA are analyzed theoretically. Two scores are
designed to measure the generalized ACs and SH detected by RePRA, and therefore
the quality of representations can be evaluated. In experiments,
representations of molecules from 10 target tasks generated by 7 pre-trained
models are analyzed. The results indicate that the state-of-the-art pre-trained
models can overcome some shortcomings of canonical Extended-Connectivity
FingerPrints (ECFP), while the correlation between the basis of the
representation space and specific molecular substructures are not explicit.
Thus, some representations could be even worse than the canonical fingerprints.
Our method enables researchers to evaluate the quality of molecular
representations generated by their proposed self-supervised pre-trained models.
And our findings can guide the community to develop better pre-training
techniques to regularize the occurrence of ACs and SH.
