  Understanding protein sequences is vital and urgent for biology, healthcare,
and medicine. Labeling approaches are expensive yet time-consuming, while the
amount of unlabeled data is increasing quite faster than that of the labeled
data due to low-cost, high-throughput sequencing methods. In order to extract
knowledge from these unlabeled data, representation learning is of significant
value for protein-related tasks and has great potential for helping us learn
more about protein functions and structures. The key problem in the protein
sequence representation learning is to capture the co-evolutionary information
reflected by the inter-residue co-variation in the sequences. Instead of
leveraging multiple sequence alignment as is usually done, we propose a novel
method to capture this information directly by pre-training via a dedicated
language model, i.e., Pairwise Masked Language Model (PMLM). In a conventional
masked language model, the masked tokens are modeled by conditioning on the
unmasked tokens only, but processed independently to each other. However, our
proposed PMLM takes the dependency among masked tokens into consideration,
i.e., the probability of a token pair is not equal to the product of the
probability of the two tokens. By applying this model, the pre-trained encoder
is able to generate a better representation for protein sequences. Our result
shows that the proposed method can effectively capture the inter-residue
correlations and improves the performance of contact prediction by up to 9%
compared to the MLM baseline under the same setting. The proposed model also
significantly outperforms the MSA baseline by more than 7% on the TAPE contact
prediction benchmark when pre-trained on a subset of the sequence database
which the MSA is generated from, revealing the potential of the sequence
pre-training method to surpass MSA based methods in general.
