VRAM Requirements
#2
by
ahmaddanyal
- opened
How much VRAM do I need to run this model?
Hi, the structure between Qwen2.5-0.5B and Qwen2-0.5B should be similar, so the results from Qwen2 could serve as a reference: https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html
If your GPU can run Minecraft, it'll probably fit with full offload. Even at FP16 :P
The model is bf16 so for model weights it will be 1GB and additional memory for inference. So indeed around 2GB of VRAM will be enough to run the model.