Pretrain of llama nvidia 4b merged with google vison encoder. Seems it needs finetuning on your dataset. As is just the base mdoel. It can identify images. However it does not follow instructions very well.

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import (
    IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN,
    DEFAULT_IM_END_TOKEN, IGNORE_INDEX
)
from llava.conversation import conv_templates, SeparatorStyle
from datetime import date
from PIL import Image
import copy
import torch
import requests
# Model and device configurations
pretrained = "mylesgoose/Llama-3.1-Minitron-4B-Llava-Nvidia-siglip-ov"
model_name = "llava_llama3"
device = "cuda"
device_map = "auto"

# Load the pretrained model
tokenizer, model, image_processor, max_length = load_pretrained_model(
    pretrained, None, model_name, device_map=device_map, attn_implementation="flash_attention_2" 
)

# Set model to evaluation mode and tie weights
model.eval()
model.tie_weights()

# Load and process the image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
date_string: str = date.today().strftime("%d %b %Y")
# Conversation template and question
conv_template = "llava_llama_3"
question = (
    DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image"
)

# Set up conversation and prepare the prompt
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()

# Tokenize the prompt and prepare input for the model
input_ids = tokenizer_image_token(
    prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt"
).unsqueeze(0).to(device)
image_sizes = [image.size]


# Generate output from the model
cont = model.generate(
    input_ids,
    images=image_tensor,
    image_sizes=image_sizes,
    do_sample=True,
    temperature=0.6,
    #pad_token_id=128004,
    #eos_token_id=None,
    max_new_tokens=131072,
)

# Decode and print the generated output
text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
print(text_outputs)

exerpt from the llava one vison repo Below.

LLaVA-NeXT: Open Large Multimodal Models

Release Notes

[2024/09/13] 🔥 🚀 LLaVA-OneVision-Chat. The new LLaVA-OV-Chat (7B/72B) significantly improves the chat experience of LLaVA-OV. 📄
[2024/08/06] 🔥 🚀 LLaVA-OneVision (OV)! The new LLaVA-OV models (0.5B/7B/72B) achieve new state-of-the-art performance across single-image, multi-image, and video benchmarks, sometimes rivaling top commercial models on 47 diverse benchmarks. 📄 Explore More:
- [Paper]: In-depth insights, new emegerging scenarios, ie, strong video understadning through task transfer from images.
- [LLaVA-OV Doc]: Model inference and evaluation guidance.
- [Scripts]: Start training models on your single-image/multi-image/video data.
[2024/07/16] 🔥 LLaVA-NeXT-Video has been upgraded. The new 32B model achieves the best open-source performance on several video benchmarks, including Video-MME. Please refer to this page for details, refer to llava_next-video_demo for demo.
[2024/06/23] 🔥 LLaVA-NeXT-Interleave is released. We utilize image-text interleaved format to unify multi-image, video, and 3D tasks in one LLM and achieve SoTA performance on a wide range of benchmarks. Check out paper, blog, and checkpoints to see new capabilities and improved performance! We have released 0.5b, 7b, and 7b-dpo models.
- An all-round LLM for multi-image, video, and 3D with strong performance [demo]
- Construct interleave training data M4-Instruct
- Construct multi-image benchmark LLaVA-Interleave Bench
[2024/05/25] 🔥 Wondering "What Else Influences Visual Instruction Tuning Beyond Data?" Our new blog summarizes empirical explorations to ablate the various design choices in improving LMMs except instruct data itself. Meanwhile, open-source the recapioned high-quality data using LLaVA-NeXT-34B on [COCO] [LCS] [CC3M].
- Architectures (LMM & Vision Encoder)
- Visual Representations (Resolution & # Tokens)
- Training Strategies (High-quality data & Trainable modules)
[2024/05/10] 🔥 LLaVA-NeXT (Stronger) models are released, with support of stronger LMM inlcuding LLama-3 (8B) and Qwen-1.5 (72B/110B) Check out [blog] and [checkpoints] to see improved performance!
[2024/05/10] 🔥 LLaVA-NeXT (Video) is released. The image-only-trained LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer. DPO training with AI feedback on videos can yield significant improvement. [Blog], [checkpoints] and [sglang]
[2024/01/30] 🔥 LLaVA-NeXT is out! With additional scaling to LLaVA-1.5, LLaVA-NeXT-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks/applications than before. Check out the blog post, and explore the demo! Models are available in Model Zoo. Training/eval data and scripts coming soon.

More
[2024/03/10] 🔥 Releasing LMMs-Eval, a highly efficient evaluation pipeline we used when developing LLaVA-NeXT. It supports the evaluation of LMMs on dozens of public datasets and allows new dataset onboarding, making the dev of new LMMs much faster. [Blog] [Codebase]
[2023/11/10] LLaVA-Plus is released: Learning to Use Tools for Creating Multimodal Agents, with LLaVA-Plus (LLaVA that Plug and Learn to Use Skills). [Project Page] [Demo] [Code] [Paper]
[2023/11/02] LLaVA-Interactive is released: Experience the future of human-AI multimodal interaction with an all-in-one demo for Image Chat, Segmentation, Generation and Editing. [Project Page] [Demo] [Code] [Paper]
[2023/10/26] 🔥 LLaVA-1.5 with LoRA achieves comparable performance as full-model finetuning, with a reduced GPU RAM requirement (ckpts, script). We also provide a doc on how to finetune LLaVA-1.5 on your own dataset with LoRA.
[2023/10/12] Check out the Korean LLaVA (Ko-LLaVA), created by ETRI, who has generously supported our research! [🤗 Demo]
[2023/10/05] 🔥 LLaVA-1.5 is out! Achieving SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods like Qwen-VL-Chat that use billion-scale data. Check out the technical report, and explore the demo! Models are available in Model Zoo. The training data and scripts of LLaVA-1.5 are released here, and evaluation scripts are released here!
[2023/09/26] LLaVA is improved with reinforcement learning from human feedback (RLHF) to improve fact grounding and reduce hallucination. Check out the new SFT and RLHF checkpoints at project [LLavA-RLHF]
[2023/09/22] LLaVA is accepted by NeurIPS 2023 as oral presentation, and LLaVA-Med is accepted by NeurIPS 2023 Datasets and Benchmarks Track as spotlight presentation.
[2023/11/06] Support Intel dGPU and CPU platforms. More details here.
[2023/10/12] LLaVA is now supported in llama.cpp with 4-bit / 5-bit quantization support!
[2023/10/11] The training data and scripts of LLaVA-1.5 are released here, and evaluation scripts are released here!
[2023/10/10] Roboflow Deep Dive: First Impressions with LLaVA-1.5.
[2023/09/20] We summarize our empirical study of training 33B and 65B LLaVA models in a note. Further, if you are interested in the comprehensive review, evolution and trend of multimodal foundation models, please check out our recent survey paper ``Multimodal Foundation Models: From Specialists to General-Purpose Assistants''.
[2023/07/19] 🔥 We release a major upgrade, including support for LLaMA-2, LoRA training, 4-/8-bit inference, higher resolution (336x336), and a lot more. We release LLaVA Bench for benchmarking open-ended visual chat with results from Bard and Bing-Chat. We also support and verify training with RTX 3090 and RTX A6000. Check out LLaVA-from-LLaMA-2, and our model zoo!
[2023/06/26] CVPR 2023 Tutorial on Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4! Please check out [Slides] [Notes] [YouTube] [Bilibli].
[2023/06/11] We released the preview for the most requested feature: DeepSpeed and LoRA support! Please see documentations here.
[2023/06/01] We released LLaVA-Med: Large Language and Vision Assistant for Biomedicine, a step towards building biomedical domain large language and vision models with GPT-4 level capabilities. Checkout the paper and page.
[2023/05/06] We are releasing LLaVA-Lighting-MPT-7B-preview, based on MPT-7B-Chat! See here for more details.
[2023/05/02] 🔥 We are releasing LLaVA-Lighting! Train a lite, multimodal GPT-4 with just $40 in 3 hours! See here for more details.
[2023/04/27] Thanks to the community effort, LLaVA-13B with 4-bit quantization allows you to run on a GPU with as few as 12GB VRAM! Try it out here.
[2023/04/17] 🔥 We released LLaVA: Large Language and Vision Assistant. We propose visual instruction tuning, towards building large language and vision models with GPT-4 level capabilities. Checkout the paper and demo.

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama-1/2 community license for LLaMA-2 and Vicuna-v1.5, Tongyi Qianwen RESEARCH LICENSE AGREEMENT and Llama-3 Research License). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

Models & Scripts

Installation

1. Clone this repository and navigate to the LLaVA folder:

git clone https://github.com/LLaVA-VL/LLaVA-NeXT
cd LLaVA-NeXT

2. Install the inference package:

conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # Enable PEP 660 support.
pip install -e ".[train]"

Project Navigation

Please checkout the following page for more inference & evaluation details.

- LLaVA-OneVision: Easy Task Transfer

LLaVA-OneVision: for demo inference. The evaluation code is in lmms-eval.

- LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild

LLaVA-NeXT-Image: for image demo inference and evaluation of stronger LMMs using lmms-eval.

- LLaVA-NeXT: A Strong Zero-shot Video Understanding Model

LLaVA-NeXT-Video: for video inference and evaluation scripts. We recommend to use LMMs-video for evaluation.

- LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models

LLaVA-NeXT-Interleave: for multi-image demo and evaluation scripts.

SGLang for SpeedUp Inference and Deployment

We use SGLang to speed up inference and deployment of LLaVA-NeXT. You could make LLaVA-NeXT as a backend API service with SGLang.

Prepare Environment: Following the instruction in the sglang

LLaVA-NeXT/OneVision

Checkout the HTTP Post/Get and SRT usage at sglang/examples/runtime/llava_onevision

LLaVA-NeXT (Video)

Launch and Run on (K) Nodes:

Go to sglang project
```
cd PATH_TO/sglang
```

First node:

bash examples/usage/llava_video/srt_example_llava_v.sh K 0 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO
(e.g. bash examples/usage/llava_video/srt_example_llava_v.sh K 0 examples/usage/llava_video/videos/Q98Z4OTh8RwmDonc.mp4 lmms-lab/LLaVA-NeXT-Video-7B-DPO 16)

Second node:

bash examples/usage/llava_video/srt_example_llava_v.sh K 1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO

The K node:

bash examples/usage/llava_video/srt_example_llava_v.sh K K-1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO

Citation

If you find it useful for your research and applications, please cite related papers/blogs using this BibTeX:

@article{li2024llava,
  title={LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models},
  author={Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan},
  journal={arXiv preprint arXiv:2407.07895},
  year={2024}
}

@misc{li2024llavanext-ablations,
    title={LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?},
    url={https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/},
    author={Li, Bo and Zhang, Hao and Zhang, Kaichen and Guo, Dong and Zhang, Yuanhan and Zhang, Renrui and Li, Feng and Liu, Ziwei and Li, Chunyuan},
    month={May},
    year={2024}
}

@misc{li2024llavanext-strong,
    title={LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild},
    url={https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/},
    author={Li, Bo and Zhang, Kaichen and Zhang, Hao and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Yuanhan and Liu, Ziwei and Li, Chunyuan},
    month={May},
    year={2024}
}

@misc{zhang2024llavanext-video,
  title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model},
  url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/},
  author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan},
  month={April},
  year={2024}
}

@misc{liu2024llavanext,
    title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},
    url={https://llava-vl.github.io/blog/2024-01-30-llava-next/},
    author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae},
    month={January},
    year={2024}
}

@misc{liu2023improvedllava,
      title={Improved Baselines with Visual Instruction Tuning}, 
      author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},
      publisher={arXiv:2310.03744},
      year={2023},
}

@misc{liu2023llava,
      title={Visual Instruction Tuning}, 
      author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
      publisher={NeurIPS},
      year={2023},
}

Acknowledgement

Vicuna: the codebase we built upon, and our base model Vicuna-13B that has the amazing language capabilities!
The LLaVA-NeXT project is currently maintained by the team along with our contributors (listed alphabetically by the first names): Bo Li, Dong Guo, Feng Li, Hao Zhang, Kaichen Zhang, Renrui Zhang, Yuanhan Zhang, led by Chunyuan Li and with the guidance and help from Haotian Liu.
The lmms-eval framework and its core contributors, including Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, and Kairui Hu, for their support on the evaluation side.

Related Projects

For future project ideas, please check out:

SEEM: Segment Everything Everywhere All at Once
Grounded-Segment-Anything to detect, segment, and generate anything by marrying Grounding DINO and Segment-Anything.