license: llama2
Tarsier Model Card
Model details
Model type: Tarsier-7b is one of the Tarsier family -- an open-source large-scale video-language models, which is designed to generate high-quality video descriptions, together with good capability of general video understanding (Tarsier-34b gains SOTA results on 6 open benchmarks). Base LLM: liuhaotian/llava-v1.6-vicuna-7b
Model date: Tarsier-7b was trained in June 2024.
Paper or resources for more information:
- github repo: https://github.com/bytedance/tarsier
- paper link: https://arxiv.org/abs/2407.00634
License
lmsys/vicuna-7b-v1.5 license.
Where to send questions or comments about the model: https://github.com/bytedance/tarsier/issues
Intended use
Primary intended uses: The primary use of Tarsier is research on large multimodal models, especially video description.
Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
Training dataset
Tarsier tasks a two-stage training strategy.
Stage-1: Multi-task Pre-training
In stage-1, we trained our model across:
- 10M diverse public datasets, such as video captioning, video question answering, action recognition, multi-image understanding, and text generation.
- 3.5M in-house data, including 2.4M high-quality video caption data similar to WebVid and 1.1M videos with object-tracking (processed on videos from Webvid and HD-VILA by object tracking tool: DEVA)
Stage-2: Multi-grained Instruction Tuning
In stage-2, we use 500K of in-house instruction tuning data, including:
- Movie clips featuring multiple shots, subjects, or events, and had annotators provide descriptions varying in length and detail, from brief motion summaries to comprehensive narratives of visual details.
- A dataset rich in camera motions, including zooming, translating, panning, and rotating.
- Video-aware creative writing, such as poems, dialogues, speeches.
Evaluation dataset
- A challenging video desription dataset: DREAM-1K
- Multi-choice VQA: MVBench, NeXT-QA and Egoschema
- Open-ended VQA: MSVD-QA, MSR-VTT-QA, ActivityNet-QA and TGIF-QA
- Video Caption: MSVD-Caption, MSRVTT-Caption, VATEX
How to Use
see https://github.com/bytedance/tarsier?tab=readme-ov-file#usage