|
---
|
|
license: llama2
|
|
---
|
|
|
|
# Tarsier Model Card
|
|
## Model details
|
|
**Model type:**
|
|
Tarsier-7b is one of the Tarsier family -- an open-source large-scale video-language models, which is designed to generate high-quality video descriptions, together with good capability of general video understanding (Tarsier-34b gains SOTA results on 6 open benchmarks). Base LLM: [liuhaotian/llava-v1.6-vicuna-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b)
|
|
|
|
**Model date:**
|
|
Tarsier-7b was trained in June 2024.
|
|
|
|
**Paper or resources for more information:**
|
|
- github repo: https://github.com/bytedance/tarsier
|
|
- paper link: https://arxiv.org/abs/2407.00634
|
|
|
|
## License
|
|
lmsys/vicuna-7b-v1.5 license.
|
|
|
|
**Where to send questions or comments about the model:**
|
|
https://github.com/bytedance/tarsier/issues
|
|
|
|
## Intended use
|
|
**Primary intended uses:**
|
|
The primary use of Tarsier is research on large multimodal models, especially video description.
|
|
|
|
**Primary intended users:**
|
|
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
|
|
|
|
## Training dataset
|
|
Tarsier tasks a two-stage training strategy.
|
|
1. Stage-1: Multi-task Pre-training
|
|
|
|
In stage-1, we trained our model across:
|
|
- 10M diverse public datasets, such as video captioning, video question answering, action recognition, multi-image understanding, and text generation.
|
|
- 3.5M in-house data, including 2.4M high-quality video caption data similar to WebVid and 1.1M videos with object-tracking (processed on videos from Webvid and HD-VILA by object tracking tool: [DEVA](https://github.com/hkchengrex/Tracking-Anything-with-DEVA))
|
|
2. Stage-2: Multi-grained Instruction Tuning
|
|
|
|
In stage-2, we use 500K of in-house instruction tuning data, including:
|
|
- Movie clips featuring multiple shots, subjects, or events, and had annotators provide descriptions varying in length and detail, from brief motion summaries to comprehensive narratives of visual details.
|
|
- A dataset rich in camera motions, including zooming, translating, panning, and rotating.
|
|
- Video-aware creative writing, such as poems, dialogues, speeches.
|
|
|
|
## Evaluation dataset
|
|
- A challenging video desription dataset: [DREAM-1K](https://huggingface.co/datasets/omni-research/DREAM-1K)
|
|
- Multi-choice VQA: [MVBench](https://huggingface.co/datasets/OpenGVLab/MVBench), [NeXT-QA](https://github.com/doc-doc/NExT-QA) and [Egoschema](https://drive.google.com/drive/folders/1SS0VVz8rML1e5gWq7D7VtP1oxE2UtmhQ)
|
|
- Open-ended VQA: [MSVD-QA](https://opendatalab.com/OpenDataLab/MSVD), [MSR-VTT-QA](https://opendatalab.com/OpenDataLab/MSR-VTT), [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa) and [TGIF-QA](https://opendatalab.com/OpenDataLab/TGIF-QA)
|
|
- Video Caption: [MSVD-Caption](https://opendatalab.com/OpenDataLab/MSVD), [MSRVTT-Caption](https://opendatalab.com/OpenDataLab/MSR-VTT), [VATEX](https://eric-xw.github.io/vatex-website/about.html)
|
|
|
|
## How to Use
|
|
see https://github.com/bytedance/tarsier?tab=readme-ov-file#usage
|
|
|
