Model Loading Error
Running your code in colab but getting error: Error loading models: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.
Thanks for the report. I wonder if the library's specifications have changed...🤔
I don't know the direct cause, so I've tried removing the suspicious parts for now.
@John6666
The error still seems to be occurring. Could you please check?
Running on cuda
Loading in NF4
Loading CLIP 📎
Loading VLM's custom vision model 📎
Loading tokenizer 🪙
Loading LLM: unsloth/Meta-Llama-3.1-8B-bnb-4bit 🤖
Error loading models: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.
It seems that accelerate and bitsandbytes were fighting. It's probably because of the version upgrade of accelerate in Colab. I think the change I just made will fix this.
https://github.com/OpenBMB/MiniCPM-o/issues/379
Thank you, it worked properly!