Instructions to use KoboldAI/GPT-J-6B-Skein with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KoboldAI/GPT-J-6B-Skein with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="KoboldAI/GPT-J-6B-Skein")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("KoboldAI/GPT-J-6B-Skein") model = AutoModelForCausalLM.from_pretrained("KoboldAI/GPT-J-6B-Skein") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use KoboldAI/GPT-J-6B-Skein with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KoboldAI/GPT-J-6B-Skein" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KoboldAI/GPT-J-6B-Skein", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/KoboldAI/GPT-J-6B-Skein
- SGLang
How to use KoboldAI/GPT-J-6B-Skein with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KoboldAI/GPT-J-6B-Skein" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KoboldAI/GPT-J-6B-Skein", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KoboldAI/GPT-J-6B-Skein" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KoboldAI/GPT-J-6B-Skein", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use KoboldAI/GPT-J-6B-Skein with Docker Model Runner:
docker model run hf.co/KoboldAI/GPT-J-6B-Skein
Enable Adventure Mode
Browse filesSince we have Janeway as a pure 6B novel model and Skein behaves the best with adventure mode enabled lets have that on by default. If people wish to use it for Novels it would give better results if they toggle to story, and otherwise they can turn it back off. This will give AID players a nicer experience, and Novel writers typically are quick to toggle it to story.
- config.json +1 -0
config.json
CHANGED
|
@@ -32,5 +32,6 @@
|
|
| 32 |
},
|
| 33 |
"torch_dtype": "float16",
|
| 34 |
"use_cache": true,
|
|
|
|
| 35 |
"vocab_size": 50400
|
| 36 |
}
|
|
|
|
| 32 |
},
|
| 33 |
"torch_dtype": "float16",
|
| 34 |
"use_cache": true,
|
| 35 |
+
"adventure": true,
|
| 36 |
"vocab_size": 50400
|
| 37 |
}
|