staghado/LightOnOCR-2-1B-Q4_K_M-GGUF

LightOnOCR-2-1B GGUF (Q4_K_M)

GGUF quantized version of lightonai/LightOnOCR-2-1B.

Files

• LightOnOCR-2-1B-Q4_K_M.gguf (378 MB) - Language model (596M parameters, Q4_K_M quantization)
• mmproj-LightOnOCR-2-1B-Q8_0.gguf (429 MB) - Vision encoder (403M parameters, Q8_0 quantization)

Usage

llama-server -hf staghado/LightOnOCR-2-1B-Q4_K_M-GGUF -c 8192 --temp 0.2 --top-k 0 --top-p 0.9

Note: The flags --temp 0.2 --top-k 0 --top-p 0.9 set the default generation parameters to match the original model.

API Example

import requests
import base64

with open('document.png', 'rb') as f:
    image_base64 = base64.b64encode(f.read()).decode()

response = requests.post('http://localhost:8000/v1/chat/completions', json={
    "model": "LightOnOCR-2-1B",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}}
        ]
    }],
    "max_tokens": 1024,
    "temperature": 0.2,
    "top_k": 0,
    "top_p": 0.9
})

print(response.json()['choices'][0]['message']['content'])

Note: This model only accepts images, no text prompts.

Creating Quantized Versions

If you want to create your own quantized GGUF files:

Prerequisites

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
python -m venv venv
source venv/bin/activate
pip install git+https://github.com/huggingface/transformers.git torch sentencepiece

Note: transformers must be installed from source until the next release includes LightOnOCR support.

Conversion Steps

1. Download original model

hf download lightonai/LightOnOCR-2-1B --repo-type model --local-dir ./models/LightOnOCR-2-1B

2. Convert language model to BF16 GGUF

python convert_hf_to_gguf.py ./models/LightOnOCR-2-1B --outtype bf16 --outfile LightOnOCR-2-1B-bf16.gguf

3. Quantize language model to Q4_K_M

llama-quantize LightOnOCR-2-1B-bf16.gguf LightOnOCR-2-1B-Q4_K_M.gguf Q4_K_M

4. Convert vision encoder to Q8_0

python convert_hf_to_gguf.py ./models/LightOnOCR-2-1B --mmproj --outtype q8_0 --outfile mmproj-LightOnOCR-2-1B-Q8_0.gguf

Notes

• Q4_K_M provides balanced compression (~8x) with minimal quality loss
• Vision encoder uses Q8_0 due to patch embedding dimension constraints
• Requires latest llama.cpp from main branch

Details

• Total: 1.01B parameters (vision: 403M + language: 596M + projector: 6M)
• Quantization: Q4_K_M (language), Q8_0 (vision)
• Tested on M3 Mac: 413 tokens/sec (prompt), 114 tokens/sec (generation)