MiaoshouAI

/

Florence-2-large-PromptGen-v1.5

like 53

Is this issue why I'm getting the following error when trying to use this model in ComfyUI Portable?

ValueError: The checkpoint you are trying to load has model type florence2 but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

I am using xformers xformers-0.0.28.post1 since I am using pytorch 2.4.1+cu124

I receive a similar error when trying to run. I am able to run Florence 2 in a separate dir+venv without issue.

(venv) ➜  MiaoshouAI git:(main) ✗ python main.py
Traceback (most recent call last):
  File "/home/user/projects/MiaoshouAI/venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1023, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/projects/MiaoshouAI/venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 725, in __getitem__
    raise KeyError(key)
KeyError: 'florence2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/projects/MiaoshouAI/main.py", line 8, in <module>
    model = AutoModelForCausalLM.from_pretrained("MiaoshouAI/Florence-2-large-PromptGen-v1.5", trust_remote_code=True).to(device)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/projects/MiaoshouAI/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/projects/MiaoshouAI/venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1025, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `florence2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

Some guidance on what I may be doing wrong would be much appreciated. Thank you.

update to the latest version of ComfyUI node to sovle the above issue.