Created Readme

README.md

@@ -0,0 +1,118 @@
+---
+library_name: transformers
+tags:
+- 4bit
+- AWQ
+- AutoAWQ
+- llama
+- llama-2
+- facebook
+- meta
+- 7b
+- quantized
+license: llama2
+pipeline_tag: text-generation
+---
+
+# Model Card for alokabhishek/Llama-2-7b-chat-hf-4bit-AWQ
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This repo contains 4-bit quantized (using AutoAWQ) model of Meta's meta-llama/Llama-2-7b-chat-hf
+
+## Model Details
+
+- Model creator: [Meta](https://huggingface.co/meta-llama)
+- Original model: [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
+
+
+### About 4 bit quantization using AutoAWQ
+
+AutoAWS github repo: [bitsandbytes github repo](https://github.com/casper-hansen/AutoAWQ/tree/main)
+
+# How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+## How to run from Python code
+
+#### First install the package
+```shell
+!pip install autoawq
+!pip install accelerate
+```
+
+#### Import 
+
+```python
+import torch
+import os
+from torch import bfloat16
+from huggingface_hub import login, HfApi, create_repo
+from transformers import AutoTokenizer, pipeline
+from awq import AutoAWQForCausalLM
+```
+
+#### Use a pipeline as a high-level helper
+
+```python
+# define the model ID
+model_id_llama = "alokabhishek/Llama-2-7b-chat-hf-4bit-AWQ"
+
+# Load model
+tokenizer_llama = AutoTokenizer.from_pretrained(model_id_llama, use_fast=True)
+model_llama = AutoAWQForCausalLM.from_quantized(model_id_llama, fuse_layer=True, trust_remote_code = False, safetensors = True)
+
+# Set up the prompt and prompt template. Change instruction as per requirements.
+prompt_llama = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
+fromatted_prompt = f'''[INST] <<SYS>> You are a helpful, and fun loving assistant. Always answer as jestfully as possible. <</SYS>> {prompt_llama} [/INST] '''
+
+tokens = tokenizer_llama(fromatted_prompt, return_tensors="pt").input_ids.cuda()
+
+# Generate output, adjust parameters as per requirements
+generation_output = model_llama.generate(tokens, do_sample=True, temperature=1.7, top_p=0.95, top_k=40, max_new_tokens=512)
+
+# Print the output
+print(tokenizer_llama.decode(generation_output[0], skip_special_tokens=True))
+
+
+```
+
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]