OpenGVLab

/

InternVL2-26B

like 114

Follow

OpenGVLab 272

Hello, thank you for your attention. For some special reason, we will restore the demo in the next few days.

@czczup I'm confused I did some visual answering with the 26b model and it performs very badly in that. The only model that passes that question are Gemini 1.5 pro/flash, gpt4-o, and Claude.

Then how InternVL2-26B was evaluated that it outperforms gpt4-v?

Thanks for your feedback. Could you give me some examples to reproduce this issue?

@czczup It was visual answering task, Can you tell me how to prompt this model to get the most accurate answer with human-like reasoning skills?

@czczup How a good prompt to this model look like?

@czczup I'm confused I did some visual answering with the 26b model and it performs very badly in that. The only model that passes that question are Gemini 1.5 pro/flash, gpt4-o, and Claude.

Then how InternVL2-26B was evaluated that it outperforms gpt4-v?

Thank you for this question.

We've achieved comparable performance to GPT-4V in many single-turn QA benchmarks. However, these results don't reflect other capabilities such as multi-turn conversation and instruction-following. (Of course, I think OpenAI also hasn't put their utmost effort into improving these academic benchmarks.)

For example, on the recently released ConvBench, which evaluates multi-turn dialogue, there is still a significant gap between our model and GPT-4V. This is mainly because most of our training data is sourced from open data. Despite our best efforts in data cleaning, the quality of our data is not as high as that of commercial companies.

As a non-profit research organization, OpenGVLab does not have the same funding as OpenAI for manually annotated data. Nevertheless, we still hope to do our best to improve the performance of open-source MLLMs.

@czczup Got it, When InternVL2-Pro is coming on HuggingFace?

We have recently open-sourced a 78B version, available at https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B. This version achieves performance similar to InternVL2-Pro on most metrics. InternVL2-Pro is still undergoing our internal lab approval process, so it might take some time before it's available.

@czczup The 78B version also didn't perform too well it can't get the answer that I get with GPT-4o, Gemini.

Thoughts?