Update README.md

README.md

@@ -1,54 +1,47 @@
----
-license: apache-2.0
-tags:
-- video LLM
----
-
-
-# Tarsier Model Card
-## Model details
-**Model type:** 
-Tarsier-34b is an open-source large-scale video-language models, which is designed to generate high-quality video descriptions, together with good capability of general video understanding (SOTA results on 6 open benchmarks).
-
-**Model date:**
-Tarsier-34b was trained in June 2024.
-
-**Paper or resources for more information:**
-- github repo: https://github.com/bytedance/tarsier
-- paper link: https://arxiv.org/abs/2407.00634
-
-## License
-NousResearch/Nous-Hermes-2-Yi-34B license.
-
-**Where to send questions or comments about the model:**
-https://github.com/bytedance/tarsier/issues
-
-## Intended use
-**Primary intended uses:**
-The primary use of Tarsier is research on large multimodal models, especially video description.
-
-**Primary intended users:**
-The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
-
-## Training dataset
-Tarsier tasks a two-stage training strategy.
-1. Stage-1: Multi-task Pre-training
-  
-    In stage-1, we trained our model across:
-    - 10M diverse public datasets, such as video captioning, video question answering, action recognition, multi-image understanding, and text generation.
-    - 3.5M in-house data, including 2.4M high-quality video caption data similar to WebVid and 1.1M videos with object-tracking (processed on videos from Webvid and HD-VILA by object tracking tool: [DEVA](https://github.com/hkchengrex/Tracking-Anything-with-DEVA))
-2. Stage-2: Multi-grained Instruction Tuning
-
-    In stage-2, we use 500K of in-house instruction tuning data, including:
-    - Movie clips featuring multiple shots, subjects, or events, and had annotators provide descriptions varying in length and detail, from brief motion summaries to comprehensive narratives of visual details.
-    - A dataset rich in camera motions, including zooming, translating, panning, and rotating.
-    - Video-aware creative writing, such as poems, dialogues, speeches.
-
-## Evaluation dataset
-- A challenging video desription dataset: [DREAM-1K](https://huggingface.co/datasets/omni-research/DREAM-1K)
-- Multi-choice VQA: [MVBench](https://huggingface.co/datasets/OpenGVLab/MVBench), [NeXT-QA](https://github.com/doc-doc/NExT-QA) and [Egoschema](https://drive.google.com/drive/folders/1SS0VVz8rML1e5gWq7D7VtP1oxE2UtmhQ)
-- Open-ended VQA: [MSVD-QA](https://opendatalab.com/OpenDataLab/MSVD), [MSR-VTT-QA](https://opendatalab.com/OpenDataLab/MSR-VTT), [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa) and [TGIF-QA](https://opendatalab.com/OpenDataLab/TGIF-QA)
-- Video Caption: [MSVD-Caption](https://opendatalab.com/OpenDataLab/MSVD), [MSRVTT-Caption](https://opendatalab.com/OpenDataLab/MSR-VTT), [VATEX](https://eric-xw.github.io/vatex-website/about.html)
-
-## How to Use
-see https://github.com/bytedance/tarsier?tab=readme-ov-file#usage
+---
+license: apache-2.0
+tags:
+- video LLM
+---
+
+
+# Tarsier Model Card
+## Model details
+**Model type:** 
+Tarsier-34b is an open-source large-scale video-language models, which is designed to generate high-quality video descriptions, together with good capability of general video understanding (SOTA results on 6 open benchmarks).
+
+**Model date:**
+Tarsier-34b was trained in June 2024.
+
+**Paper or resources for more information:**
+- github repo: https://github.com/bytedance/tarsier
+- paper link: https://arxiv.org/abs/2407.00634
+
+## License
+NousResearch/Nous-Hermes-2-Yi-34B license.
+
+**Where to send questions or comments about the model:**
+https://github.com/bytedance/tarsier/issues
+
+## Intended use
+**Primary intended uses:**
+The primary use of Tarsier is research on large multimodal models, especially video description.
+
+**Primary intended users:**
+The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
+
+## Training dataset
+Tarsier tasks a two-stage training strategy.
+- Stage-1: Multi-task Pre-training on 13M data
+- Stage-2: Multi-grained Instruction Tuning on 500K data
+  
+In both stages, we freeze ViT and train all the parameters of projection layer and LLM.
+
+## Evaluation dataset
+- A challenging video desription dataset: [DREAM-1K](https://huggingface.co/datasets/omni-research/DREAM-1K)
+- Multi-choice VQA: [MVBench](https://huggingface.co/datasets/OpenGVLab/MVBench), [NeXT-QA](https://github.com/doc-doc/NExT-QA) and [Egoschema](https://drive.google.com/drive/folders/1SS0VVz8rML1e5gWq7D7VtP1oxE2UtmhQ)
+- Open-ended VQA: [MSVD-QA](https://opendatalab.com/OpenDataLab/MSVD), [MSR-VTT-QA](https://opendatalab.com/OpenDataLab/MSR-VTT), [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa) and [TGIF-QA](https://opendatalab.com/OpenDataLab/TGIF-QA)
+- Video Caption: [MSVD-Caption](https://opendatalab.com/OpenDataLab/MSVD), [MSRVTT-Caption](https://opendatalab.com/OpenDataLab/MSR-VTT), [VATEX](https://eric-xw.github.io/vatex-website/about.html)
+
+## How to Use
+see https://github.com/bytedance/tarsier?tab=readme-ov-file#usage