microsoft

/

Florence-2-large

like 1.62k

Follow

Microsoft 13.9k

Have you solved this problem?

• 1 here

I fixed it by directly initializing the model and loading the checkpoint, instead of relying on transformers.

@Xsong123 Could you please share the code how did you initialize it directly? I tried a few options but they fails with some florence runtime errors

I faced similar issue, but for microsoft/Florence-2-base with transformers==4.51.3. I resolved this by loading model from pretrained checkpoint (file pytorch_model.bin exists) and saving using save_pretrained. New model directory contains not pytorch_model.bin, but model.safetensors. After that model loading is very fast.

Code to reproduce

pip install transformers==4.51.3 timy

import os
from transformers import AutoModelForCausalLM, AutoProcessor
import timy
import json


def load_model(model_id: str):
    florence = AutoModelForCausalLM.from_pretrained(
        model_id,
        trust_remote_code=True,
    ).eval()
    florence_processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    return florence, florence_processor


def save_model_and_processor(
    florence: AutoModelForCausalLM, florence_processor: AutoProcessor, model_path: str
):
    florence.save_pretrained(model_path)

    florence_processor.save_pretrained(model_path)
    # fix because after saving model, model_type for vision_config is lost
    with open(os.path.join(model_path, "config.json"), "r") as f:
        config = json.load(f)
    config["vision_config"]["model_type"] = "davit"
    with open(os.path.join(model_path, "config.json"), "w") as f:
        json.dump(config, f)


if __name__ == "__main__":
    with timy.Timer("Prepare standalone model"):
        model, processor = load_model("microsoft/Florence-2-base")

    new_model_path = "new_model"
    with timy.Timer("Save model and processor"):
        save_model_and_processor(model, processor, new_model_path)
    with timy.Timer("Load new model and processor"):
        model, processor = load_model(new_model_path)

Logs:

Prepare standalone model 83.104714 seconds
Save model and processor 3.589017 seconds
Load new model and processor 1.996852 seconds

@tonny123321 Sorry for the late reply — it's been a while since I worked on this. I can't quite remember where I saved my code. If I recall correctly, I manually instantiated the model based on the official Microsoft/Florence-2-large model files, and then loaded the weights using load_state_dict directly. You could try that approach. Alternatively, you might want to follow @MateuszW 's suggestion above — that could be a more straightforward way.

@tonny123321 I found my code today, here it is below. Hope it helps!

from Florence_2_large.configuration_florence2 import Florence2Config
from Florence_2_large.modeling_florence2 import Florence2ForConditionalGeneration
import torch
from transformers import AutoProcessor
from accelerate import Accelerator

FLORENCE2_MODEL_ID = "Florence_2_large"
config_path = "./Florence_2_large/config.json"
pretrained_model_path = "./Florence_2_large/pytorch_model.bin"
florence2_config = Florence2Config.from_pretrained(config_path)
florence2_model_raw = Florence2ForConditionalGeneration(florence2_config)
florence2_model_raw.load_state_dict(torch.load(pretrained_model_path, map_location="cpu"))
florence2_model_raw = florence2_model_raw.to(torch.float16)  # Convert model to float16 before preparing
florence2_processor = AutoProcessor.from_pretrained(FLORENCE2_MODEL_ID, trust_remote_code=True)

# Prepare the model - handles device placement and DDP wrapping if needed
accelerator = Accelerator()
florence2_model = accelerator.prepare(florence2_model_raw)
accelerator.print("Florence-2 model prepared.")