  Bridging the exponentially growing gap between the numbers of unlabeled and
labeled protein sequences, several studies adopted semi-supervised learning for
protein sequence modeling. In these studies, models were pre-trained with a
substantial amount of unlabeled data, and the representations were transferred
to various downstream tasks. Most pre-training methods solely rely on language
modeling and often exhibit limited performance. In this paper, we introduce a
novel pre-training scheme called PLUS, which stands for Protein sequence
representations Learned Using Structural information. PLUS consists of masked
language modeling and a complementary protein-specific pre-training task,
namely same-family prediction. PLUS can be used to pre-train various model
architectures. In this work, we use PLUS to pre-train a bidirectional recurrent
neural network and refer to the resulting model as PLUS-RNN. Our experiment
results demonstrate that PLUS-RNN outperforms other models of similar size
solely pre-trained with the language modeling in six out of seven widely used
protein biology tasks. Furthermore, we present the results from our qualitative
interpretation analyses to illustrate the strengths of PLUS-RNN. PLUS provides
a novel way to exploit evolutionary relationships among unlabeled proteins and
is broadly applicable across a variety of protein biology tasks. We expect that
the gap between the numbers of unlabeled and labeled proteins will continue to
grow exponentially, and the proposed pre-training method will play a larger
role.
