Best model for SD?
#10
by
darkstar3537
- opened
What'd be the best model to use with this for speculative decoding? My first though is a Qwen2.5 based small model.
What'd be the best model to use with this for speculative decoding? My first though is a Qwen2.5 based small model.
@darkstar3537 If you can run both at the same time on your system, I would recommend Qwen/Qwen2.5-7B-Instruct but if you're on a resource budget, there are 3b, 1.5b, and 500m variants to choose from as well.