Create README.md

README.md

@@ -0,0 +1,61 @@
+---
+license: apache-2.0
+language:
+- en
+inference: false
+---
+# Model Card for TinyMixtral-x8-Clonebase-7b
+This model is based on [TinyLlama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T), converted to a mistral model, and then placed the clone in mixtral.  
+**This model was created experimentally for training a small mixtral.**  
+
+# How it was made
+First, since tinyllama is an llama model, I converted it to a mistral model.  
+  
+After that, I cloned the FFN part and made it experts.
+Since they are all the same tensor, the performance does not change.
+All gates have the same value.
+
+# How To Convert
+use colab cpu-high-memory.  
+This model was created with experts=8, but since it is a clone, you can create as many experts as you like.  
+
+[tinyllama_to_mixtral_clonebase.ipynb](https://huggingface.co/mmnga/TinyMixtral-x8-Clonebase-7b)
+
+# Usage
+~~~python
+pip install transformers --upgrade
+pip install flash_attn
+~~~
+
+~~~python
+from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM
+import torch
+
+model_name_or_path = "mmnga/TinyMixtral-x8-Clonebase-7b"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+model = MixtralForCausalLM.from_pretrained(model_name_or_path, device_map="auto")
+
+# set num_experts_per_tok 1 or 2 ?
+model.config.num_experts_per_tok = 2
+
+# message
+messages = [
+    {"role": "user", "content": "Tell me what's for dinner tonight."},
+]
+
+with torch.no_grad():
+    token_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
+    output_ids = model.generate(
+        token_ids.to(model.device),
+        temperature=0.5,
+        do_sample=True,
+        top_p=0.95,
+        top_k=40,
+        max_new_tokens=128,
+        repetition_penalty=1.5
+    )
+output = tokenizer.decode(output_ids[0][token_ids.size(1) :])
+print(output)
+
+~~~