  We report a flexible language-model based deep learning strategy, applied
here to solve complex forward and inverse problems in protein modeling, based
on an attention neural network that integrates transformer and graph
convolutional architectures in a causal multi-headed graph mechanism, to
realize a generative pretrained model. The model is applied to predict
secondary structure content (per-residue level and overall content), protein
solubility, and sequencing tasks. Further trained on inverse tasks, the model
is rendered capable of designing proteins with these properties as target
features. The model is formulated as a general framework, completely
prompt-based, and can be adapted for a variety of downstream tasks. We find
that adding additional tasks yields emergent synergies that the model exploits
in improving overall performance, beyond what would be possible by training a
model on each dataset alone. Case studies are presented to validate the method,
yielding protein designs specifically focused on structural proteins, but also
exploring the applicability in the design of soluble, antimicrobial
biomaterials. While our model is trained to ultimately perform 8 distinct
tasks, with available datasets it can be extended to solve additional problems.
In a broader sense, this work illustrates a form of multiscale modeling that
relates a set of ultimate building blocks (here, byte-level utf8 characters
that define the nature of the physical system at hand) to complex output. This
materiomic scheme captures complex emergent relationships between universal
building block and resulting properties via a synergizing learning capacity to
express a set of potentialities embedded in the knowledge used in training, via
the interplay of universality and diversity.
