gguf model?

#1
by segmond - opened

Can you upload a gguf model?

q8 uploadig right now (see https://huggingface.co/volker-mauel/Kimi-Dev-72B-GGUF - might take another 10m)

f32, f16, tq1_0 and tq2_0 will be there in the next 30m-1h

@volker-mauel

f32, f16, tq1_0 and tq2_0 will be there in the next 30m-1h

Hey wait, tq1_0 and tq2_0 are for ternary models only!! don't upload those they might not even work! See @compilade s note here: https://www.reddit.com/r/LocalLLaMA/comments/1la1v4d/comment/mxioh75/

I'd suggest you try like q4_K or something as a better quant choice.

Anyway, bullerwins has some usable models up now!

Cheers!

Sign up or log in to comment