keras/esm2_t12_35M

This is a ESM model uploaded using the KerasHub library and can be used with JAX, TensorFlow, and PyTorch backends. This model is related to a MaskedPLM task.

Model config:

• name: esm_backbone
• trainable: True
• vocabulary_size: 33
• num_layers: 12
• num_heads: 20
• hidden_dim: 480
• intermediate_dim: 1920
• dropout: 0.0
• max_wavelength: 10000
• use_bias: True
• activation: gelu
• layer_norm_eps: 1e-05
• use_pre_layer_norm: False
• position_embedding_type: rotary
• max_sequence_length: 1026
• pad_token_id: 1

This model card has been generated automatically and should be completed by the model author. See Model Cards documentation for more information.