  Automated computational analysis of the vast chemical space is critical for
numerous fields of research such as drug discovery and material science.
Representation learning techniques have recently been employed with the primary
objective of generating compact and informative numerical expressions of
complex data. One approach to efficiently learn molecular representations is
processing string-based notations of chemicals via natural language processing
(NLP) algorithms. Majority of the methods proposed so far utilize SMILES
notations for this purpose; however, SMILES is associated with numerous
problems related to validity and robustness, which may prevent the model from
effectively uncovering the knowledge hidden in the data. In this study, we
propose SELFormer, a transformer architecture-based chemical language model
that utilizes a 100% valid, compact and expressive notation, SELFIES, as input,
in order to learn flexible and high-quality molecular representations.
SELFormer is pre-trained on two million drug-like compounds and fine-tuned for
diverse molecular property prediction tasks. Our performance evaluation has
revealed that, SELFormer outperforms all competing methods, including graph
learning-based approaches and SMILES-based chemical language models, on
predicting aqueous solubility of molecules and adverse drug reactions. We also
visualized molecular representations learned by SELFormer via dimensionality
reduction, which indicated that even the pre-trained model can discriminate
molecules with differing structural properties. We shared SELFormer as a
programmatic tool, together with its datasets and pre-trained models. Overall,
our research demonstrates the benefit of using the SELFIES notations in the
context of chemical language modeling and opens up new possibilities for the
design and discovery of novel drug candidates with desired features.
