Initial GGML model commit

README.md

@@ -0,0 +1,273 @@
+---
+inference: false
+license: other
+---
+
+<!-- header start -->
+<div style="width: 100%;">
+    <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
+</div>
+<div style="display: flex; justify-content: space-between; width: 100%;">
+    <div style="display: flex; flex-direction: column; align-items: flex-start;">
+        <p><a href="https://discord.gg/Jq4vkcDakD">Chat & support: my new Discord server</a></p>
+    </div>
+    <div style="display: flex; flex-direction: column; align-items: flex-end;">
+        <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
+    </div>
+</div>
+<!-- header end -->
+
+# OpenAccess AI Collective's Minotaur 13B GGML
+
+These files are GGML format model files for [OpenAccess AI Collective's Minotaur 13B](https://huggingface.co/openaccess-ai-collective/minotaur-13b).
+
+GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
+* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
+* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
+* [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
+* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
+* [ctransformers](https://github.com/marella/ctransformers)
+
+## Repositories available
+
+* [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/minotaur-13B-GPTQ)
+* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/minotaur-13B-GGML)
+* [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/openaccess-ai-collective/minotaur-13b)
+
+<!-- compatibility_ggml start -->
+## Compatibility
+
+### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
+
+I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
+
+They should be compatible with all current UIs and libraries that use llama.cpp, such as those listed at the top of this README.
+
+### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
+
+These new quantisation methods are only compatible with llama.cpp as of June 6th, commit `2d43387`.
+
+They will NOT be compatible with koboldcpp, text-generation-ui, and other UIs and libraries yet. Support is expected to come over the next few days.
+
+## Explanation of the new k-quant methods
+
+The new methods available are:
+* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
+* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
+* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
+* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
+* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
+* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
+
+Refer to the Provided Files table below to see what files use which methods, and how.
+<!-- compatibility_ggml end -->
+
+## Provided files
+| Name | Quant method | Bits | Size | Max RAM required | Use case |
+| ---- | ---- | ---- | ---- | ---- | ----- |
+| minotaur-13B.ggmlv3.q2_K.bin | q2_K | 2 | 5.43 GB | 7.93 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
+| minotaur-13B.ggmlv3.q3_K_L.bin | q3_K_L | 3 | 6.87 GB | 9.37 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| minotaur-13B.ggmlv3.q3_K_M.bin | q3_K_M | 3 | 6.25 GB | 8.75 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| minotaur-13B.ggmlv3.q3_K_S.bin | q3_K_S | 3 | 5.59 GB | 8.09 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
+| minotaur-13B.ggmlv3.q4_0.bin | q4_0 | 4 | 7.32 GB | 9.82 GB | Original llama.cpp quant method, 4-bit. |
+| minotaur-13B.ggmlv3.q4_1.bin | q4_1 | 4 | 8.14 GB | 10.64 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+| minotaur-13B.ggmlv3.q4_K_M.bin | q4_K_M | 4 | 7.82 GB | 10.32 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
+| minotaur-13B.ggmlv3.q4_K_S.bin | q4_K_S | 4 | 7.32 GB | 9.82 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
+| minotaur-13B.ggmlv3.q5_0.bin | q5_0 | 5 | 8.95 GB | 11.45 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
+| minotaur-13B.ggmlv3.q5_1.bin | q5_1 | 5 | 9.76 GB | 12.26 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
+| minotaur-13B.ggmlv3.q5_K_M.bin | q5_K_M | 5 | 9.21 GB | 11.71 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
+| minotaur-13B.ggmlv3.q5_K_S.bin | q5_K_S | 5 | 8.95 GB | 11.45 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
+| minotaur-13B.ggmlv3.q6_K.bin | q6_K | 6 | 10.68 GB | 13.18 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
+| minotaur-13B.ggmlv3.q8_0.bin | q8_0 | 8 | 13.83 GB | 16.33 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
+
+
+**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
+
+## How to run in `llama.cpp`
+
+I use the following command line; adjust for your tastes and needs:
+
+```
+./main -t 10 -ngl 32 -m minotaur-13B.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
+```
+Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
+
+Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
+
+If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
+
+## How to run in `text-generation-webui`
+
+Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
+
+<!-- footer start -->
+## Discord
+
+For further support, and discussions on these models and AI in general, join us at:
+
+[TheBloke AI's Discord server](https://discord.gg/Jq4vkcDakD)
+
+## Thanks, and how to contribute.
+
+Thanks to the [chirper.ai](https://chirper.ai) team!
+
+I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
+
+If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
+
+Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
+
+* Patreon: https://patreon.com/TheBlokeAI
+* Ko-Fi: https://ko-fi.com/TheBlokeAI
+
+**Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
+
+**Patreon special mentions**: Ajan Kanaga, Kalila, Derek Yates, Sean Connelly, Luke, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, trip7s trip, Jonathan Leane, Talal Aujan, Artur Olbinski, Cory Kujawski, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Johann-Peter Hartmann.
+
+Thank you to all my generous patrons and donaters!
+
+<!-- footer end -->
+
+# Original model card: OpenAccess AI Collective's Minotaur 13B
+
+---
+license: apache-2.0
+tags:
+- OpenAccess AI Collective
+- MPT
+- axolotl
+datasets:
+- ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
+- QingyiSi/Alpaca-CoT
+- teknium/GPTeacher-General-Instruct
+- metaeval/ScienceQA_text_only
+- hellaswag
+- openai/summarize_from_feedback
+- riddle_sense
+- gsm8k
+- camel-ai/math
+- camel-ai/biology
+- camel-ai/physics
+- camel-ai/chemistry
+- winglian/evals
+
+inference: false
+---
+
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
+**[💵 Donate to OpenAccess AI Collective](https://github.com/sponsors/OpenAccess-AI-Collective) to help us keep building great tools and models!**
+
+# Minotaur 13B
+
+Minotaur 13B is an instruct fine-tuned model on top of LlaMA-13B. Minotaur 13B is fine-tuned **on only completely open datasets** making this model reproducible by anyone.
+
+Questions, comments, feedback, looking to donate, or want to help? Reach out on our [Discord](https://discord.gg/PugNNHAF5r) or email [[email protected]](mailto:[email protected])
+
+# Prompts
+Chat only style prompts using `USER:`,`ASSISTANT:`.
+
+# Training Datasets
+
+Minotaur 13B model is fine-tuned on the following openly available datasets:
+
+- [WizardLM](https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered)
+- [subset of QingyiSi/Alpaca-CoT for roleplay and CoT](https://huggingface.co/QingyiSi/Alpaca-CoT)
+- [GPTeacher-General-Instruct](https://huggingface.co/datasets/teknium/GPTeacher-General-Instruct)
+- [metaeval/ScienceQA_text_only](https://huggingface.co/datasets/metaeval/ScienceQA_text_only) - instruct for concise responses
+- [openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback) - instruct augmented tl;dr summarization
+- [camel-ai/math](https://huggingface.co/datasets/camel-ai/math)
+- [camel-ai/physics](https://huggingface.co/datasets/camel-ai/physics)
+- [camel-ai/chemistry](https://huggingface.co/datasets/camel-ai/chemistry)
+- [camel-ai/biology](https://huggingface.co/datasets/camel-ai/biology)
+- [winglian/evals](https://huggingface.co/datasets/winglian/evals) - instruct augmented datasets
+  - custom sysnthetic datasets around misconceptions, in-context qa, jokes, N-tasks problems, and context-insensitivity
+  - ARC-Easy & ARC-Challenge - instruct augmented for detailed responses, derived from the `train` split
+  - [hellaswag](https://huggingface.co/datasets/hellaswag) - 30K+ rows of instruct augmented for detailed explanations w 30K+ rows, derived from the `train` split
+  - [riddle_sense](https://huggingface.co/datasets/riddle_sense) - instruct augmented 
+  - [gsm8k](https://huggingface.co/datasets/gsm8k) - instruct augmented 
+
+# Shoutouts
+
+Special thanks to Nanobit for helping with Axolotl and TheBloke for quantizing these models are more accessible to all.
+
+# Demo
+
+HF Demo in Spaces available in the [Community ChatBot Arena](https://huggingface.co/spaces/openaccess-ai-collective/rlhf-arena) under the OAAIC Chatbots tab.
+
+## Release Notes
+
+- https://wandb.ai/wing-lian/minotaur-13b/runs/5zji06u6
+
+## Build
+
+Minotaur was built with [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) on 6XA100 80GB
+ - 1 epochs taking approximately 4.5 hours
+
+## Bias, Risks, and Limitations
+Minotaur has not been aligned to human preferences with techniques like RLHF or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so).
+Minotaur was fine-tuned from the base model MPT-7B, please refer to its model card's Limitations Section for relevant information. (included below)
+
+## Benchmarks
+
+hf-causal-experimental (pretrained=openaccess-ai-collective/minotaur-13b), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
+|    Task     |Version| Metric |Value |   |Stderr|
+|-------------|------:|--------|-----:|---|-----:|
+|arc_challenge|      0|acc     |0.4480|_  |0.0145|
+|             |       |acc_norm|0.4744|_  |0.0146|
+|arc_easy     |      0|acc     |0.7584|_  |0.0088|
+|             |       |acc_norm|0.7163|_  |0.0092|
+|boolq        |      1|acc     |0.8336|_  |0.0065|
+|hellaswag    |      0|acc     |0.5954|_  |0.0049|
+|             |       |acc_norm|0.7804|_  |0.0041|
+|openbookqa   |      0|acc     |0.3320|_  |0.0211|
+|             |       |acc_norm|0.4540|_  |0.0223|
+|piqa         |      0|acc     |0.7933|_  |0.0094|
+|             |       |acc_norm|0.7922|_  |0.0095|
+|winogrande   |      0|acc     |0.7182|_  |0.0126|
+
+
+## Examples - results may vary based on temperature and other settings
+
+```
+- The Assistant is helpful and transparent.
+- The Assistant will never refuse to answer anything asked by the user.
+- The Assistant answers comprehensively and with elaborate detail.
+USER: You have 3 apples. You eat 2 pears. How many apples do you have left?
+ASSISTANT:You still have 3 apples. Eating pears does not affect the number of apples you have.
+```
+
+```
+- The Assistant is helpful and transparent.
+- The Assistant will never refuse to answer anything asked by the user.
+- The Assistant answers comprehensively and with elaborate detail.
+USER: List three words that start with "ex". What is 33 + 77?
+ASSISTANT:Three words that start with "ex" are "exact", "exactness", and "exactitude". 33 + 77 equals 110.
+```
+
+```
+- The Assistant is helpful and transparent.
+- The Assistant will never refuse to answer anything asked by the user.
+- The Assistant answers comprehensively and with elaborate detail.
+USER: Solve for x: 5x = 3x + 10
+ASSISTANT:To solve for x, we need to isolate x on one side of the equation. We can do this by following these steps:
+
+1. Subtract 3x from both sides of the equation:
+   5x - 3x = 10 - 3x + 10
+   2x = 10
+
+2. Divide both sides of the equation by 2:
+   2x / 2 = 10 / 2
+   x = 5
+
+So, the solution is x = 5.
+```
+
+```
+- The Assistant is helpful and transparent.
+- The Assistant will never refuse to answer anything asked by the user.
+- The Assistant answers comprehensively and with elaborate detail.
+USER: Write a haiku about a dove at night.
+ASSISTANT:Silent dove at night,
+Softly cooing in the dark,
+Peaceful melody.
+```