Add files using upload-large-folder tool

README.md

@@ -0,0 +1,97 @@
+---
+{}
+---
+
+# Model Card for Dayhoff
+
+In this work, we combined genomic-derived protein sequences, metagenomics, structure-based synthetic sequences, and MSAs to create the Dayhoff Atlas of protein data and language models. 
+We first created a large-scale natural protein dataset, GigaRef,  by combining and reclustering sequences from metagenomic databases with UniRef100. With 3.3B sequences in 1.7B clusters, GigaRef is the largest open dataset of natural proteins to date. 
+
+To infuse the benefits of protein structure information into sequence space, we generated the first large-scale structure-based synthetic dataset, called BackboneRef, by sampling 240,830 backbone structures from a structure-based generative model and then using them to design a total of 46M synthetic sequences.
+Using UniRef, GigaRef, BackboneRef, and 16M MSAs from OpenProteinSet, we then trained the Dayhoff series of PLMs, which use a a hybrid state-space-model (SSM) and transformer architecture along with a mixture-of-experts (MoE) mechanism to enable the long context lengths needed to combine single sequences and MSAs at scale. 
+Dayhoff models make accurate zero-shot predictions of mutations effects, generate sequences conditioned on aligned or unaligned homologs, and generate shorter Cas9s that preserve the functional domain architecture. 
+
+Larger models, metagenomic sequences, and structure-based augmentation all increased the expression rates of unconditional generations in E. coli. 
+Finally, we generated, characterized, and release 16M synthetic sequences as DayhoffRef
+
+Dayhoff is described in this [preprint](preprint); if you use the code from this repository or the results, please cite the preprint.
+
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** Kevin K. Yang, Sarah Alamdari, Alex J. Lee, Kaeli Kaymak-Loveless, Samir Char, Garyk Brixi, Carles Domingo-Enrich, Chentong Wang, Suyue Lyu, Nicolo Fusi, Neil Tenenholtz, Ava P. Amini
+- **Model type:** Hybrid state-space-model transformer architecture with mixture-of-experts
+- **License:** MIT
+
+### Model Sources
+
+- **Repository:** https://github.com/microsoft/dayhoff
+
+## Uses
+
+### Downstream Use
+
+* Protein Language Model Training: Training protein language models, to generate new protein sequences, predict mutation effects, and design functional proteins.
+* Zero-shot Prediction: Predicting the functional impact of mutations.
+* Sequence Generation: Generating new protein sequences unconditionally, or conditioned on homologs for designing proteins with desired properties.
+* Synthetic Sequence Generation: Exploring novel protein structures and functions using synthetic sequences.
+
+
+## Bias, Risks, and Limitations
+
+The [software/model] described in this repository is provided for research and development use only. The [software/model] is not intended for use in clinical decision-making or for any other clinical use, and the performance of model for clinical use has not been established. You bear sole responsibility for any use of this [software/model], including incorporation into any product intended for clinical use. 
+
+
+## How to Get Started with the Model
+
+Sample protein generation code:
+
+```py
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
+
+set_seed(0)
+torch.set_default_device("cuda")
+
+model = AutoModelForCausalLM.from_pretrained('microsoft/Dayhoff-170m-UR90')
+tokenizer = AutoTokenizer.from_pretrained('microsoft/Dayhoff-170m-UR90', trust_remote_code=True)
+
+
+inputs = tokenizer(tokenizer.bos_token, return_tensors="pt", return_token_type_ids=False)
+
+outputs = model.generate(inputs['input_ids'],max_length=50,do_sample=True)
+sequence = tokenizer.batch_decode(outputs,skip_special_tokens=True)
+print(sequence)
+```
+
+For detailed instructions on package usage, please refer to the README in model repo.
+
+## Evaluation
+
+### Results
+
+Dayhoff models make accurate zero-shot predictions of mutation effects, generate sequences conditioned on aligned or unaligned homologs, and generate shorter Cas9s that preserve the functional domain architecture. Larger models, metagenomic sequences, and structure-based augmentation all increased the expression rates of unconditional generations in E. coli 
+
+## Technical Specifications 
+
+### Compute Infrastructure
+
+* 170M-parameter models: trained on 8 NVIDIA A100 or 8 NVIDIA H100 GPUs using Distributed Data Parallel.
+* 3B-parameter models: trained on 176 NVIDIA H100 GPUs using Fully Sharded Data Parallel in hybrid-shard mode.
+
+
+## Citation
+
+**BibTeX:**
+If you use this model in your work, please cite it as follows:
+
+<ADD INFO>
+
+
+## Model Card Authors
+
+Samir Char, Sarah A. Alamdari
+

config.json

@@ -0,0 +1,48 @@
+{
+  "_name_or_path": "ai21labs/Jamba-v0.1",
+  "architectures": [
+    "JambaForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "attn_layer_offset": 4,
+  "attn_layer_period": 8,
+  "auto_map": {
+    "AutoConfig": "ai21labs/Jamba-v0.1--configuration_jamba.JambaConfig",
+    "AutoModel": "ai21labs/Jamba-v0.1--modeling_jamba.JambaModel",
+    "AutoModelForCausalLM": "ai21labs/Jamba-v0.1--modeling_jamba.JambaForCausalLM",
+    "AutoModelForSequenceClassification": "ai21labs/Jamba-v0.1--model.JambaForSequenceClassification"
+  },
+  "bos_token_id": 29,
+  "eos_token_id": 27,
+  "expert_layer_offset": 1,
+  "expert_layer_period": 2,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "mamba_conv_bias": true,
+  "mamba_d_conv": 4,
+  "mamba_d_state": 16,
+  "mamba_dt_rank": 16,
+  "mamba_expand": 2,
+  "mamba_proj_bias": false,
+  "max_position_embeddings": 262144,
+  "model_type": "jamba",
+  "num_attention_heads": 16,
+  "num_experts": 16,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 8,
+  "num_logits_to_keep": 1,
+  "output_router_logits": true,
+  "pad_token_id": 30,
+  "rms_norm_eps": 1e-06,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.4",
+  "use_cache": false,
+  "use_mamba_kernels": true,
+  "vocab_size": 40
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 29,
+  "eos_token_id": 27,
+  "pad_token_id": 30,
+  "transformers_version": "4.42.4",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eec6720e94710b5229b21e1d902e0de47a323dd32b3c9f6627828f29f1b7d119
+size 681305728

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "bos_token": {
+    "content": "@",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "*",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "#",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "!",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "/",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "27": {
+      "content": "*",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "#",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "@",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "!",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "/",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenizers.ProteinTokenizer",
+      null
+    ]
+  },
+  "bos_token": "@",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "*",
+  "mask_token": "#",
+  "model_max_length": 2048,
+  "pad_token": "!",
+  "sep_token": "/",
+  "tokenizer_class": "ProteinTokenizer"
+}

tokenizers.py

@@ -0,0 +1,126 @@
+from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer
+from typing import List, Optional, Union
+import os
+
+MASK = "#"
+MSA_PAD = "!"
+UL_ALPHABET_PLUS = "ACDEFGHIKLMNPQRSTVWYBZXJOU-*#@!/[]{}"
+MSA_AAS = "ACDEFGHIKLMNPQRSTVWYBZXJOU-"
+GAP = "-"
+START = "@"
+STOP = "*"
+SEP = "/"
+END_AL = "]"
+END_UL = "}"
+START_AL = "["
+START_UL = "{"
+
+class ProteinTokenizer(PreTrainedTokenizer):
+
+    def __init__(
+        self,
+        protein_alphabet: str = UL_ALPHABET_PLUS,
+        model_max_length: int = 2048,
+        pad_token=MSA_PAD,
+        mask_token=MASK,
+        all_aas=MSA_AAS,
+        gap_token=GAP,
+        bos_token=START,
+        eos_token=STOP,
+        sep_token=SEP,
+        **kwargs
+    ):
+        """Character tokenizer for Hugging Face transformers.
+
+        model_max_length (int): Model maximum sequence length.
+        """
+        self.alphabet = list("".join(protein_alphabet))
+        self.all_aas = list("".join(all_aas))
+        self.a_to_i = {u: i for i, u in enumerate(self.alphabet)}
+        self.i_to_a = {i: u for i, u in enumerate(self.alphabet)}
+        self.gap_token = gap_token
+
+        
+        bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
+        eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
+        sep_token = AddedToken(sep_token, lstrip=False, rstrip=False) if isinstance(sep_token, str) else sep_token
+        mask_token = AddedToken(mask_token, lstrip=False, rstrip=False) if isinstance(mask_token, str) else mask_token 
+        pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
+        gap_token = AddedToken(gap_token, lstrip=False, rstrip=False) if isinstance(gap_token, str) else gap_token
+
+        super().__init__(
+            pad_token=pad_token,
+            mask_token=mask_token,
+            eos_token=eos_token,
+            bos_token=bos_token,
+            sep_token=sep_token,
+            model_max_length=model_max_length,
+            **kwargs
+        )
+
+    @property
+    def vocab_size(self):
+        return len(self.alphabet)
+    
+    @property
+    def gap_token_id(self):
+        return self.convert_tokens_to_ids(self.gap_token)
+
+    def get_vocab(self):
+        return self.a_to_i
+
+    def _tokenize(self, text: str) -> List[str]:
+        return list(text)
+
+    def _convert_token_to_id(self, token) -> int:
+        return self.a_to_i[token]
+
+    def _convert_id_to_token(self, index) -> str:
+        return self.i_to_a[index]
+
+    def convert_tokens_to_string(self, tokens):
+        return "".join(tokens)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        result = token_ids_0
+        if token_ids_1 is not None:
+            raise NotImplementedError("This tokenizer does not support two sequences")
+        return result
+
+    def get_special_tokens_mask(
+        self,
+        token_ids_0: List[int],
+        token_ids_1: Optional[List[int]] = None,
+        already_has_special_tokens: bool = False,
+    ) -> List[int]:
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=True,
+            )
+
+        result = [0] * len(token_ids_0)
+        if token_ids_1 is not None:
+            raise NotImplementedError("This tokenizer does not support two sequences")
+
+        return result
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Identifies the type of token. 0 for the first sentence, 1 for the second sentence if it exists
+        """
+
+        result = len(token_ids_0) * [0]
+
+        if token_ids_1 is not None:
+            raise NotImplementedError("This tokenizer does not support two sequences")
+        return result
+
+    def save_pretrained(self, save_directory: Union[str, os.PathLike], **kwargs):
+        super().save_pretrained(save_directory, **kwargs)
+    
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None):
+        return ()