microsoft

/

Phi-3-vision-128k-instruct

like 926

Follow

Microsoft 4,773

@abrakaa Thanks for your interest in our model. We may open-source the finetuning script within the near future.

@donniems We need it soon :)

@donniems looking forward to it

@donniems Need it now! Really looking forward to it!

@donniems Can’t wait to finetune on my own data!

@abrakaa Thanks for your interest in our model. We may open-source the finetuning script within the near future.

I wrote a finetuning script for Phi3-V: https://github.com/GaiZhenbiao/Phi3V-Finetuning/tree/main , enjoy 😉

here you go https://wandb.ai/byyoung3/mlnews3/reports/How-to-fine-tune-Phi-3-Vision-on-a-custom-dataset--Vmlldzo4MTEzMTg3

https://github.com/2U1/Phi3-Vision-ft

I've made a code that has option to fine-tune full module (including vision model) like llava-1.6 !

Thank you all your interest in Phi-3 Vision model.
This is the finetuning recipe https://github.com/microsoft/Phi-3CookBook/blob/main/md/04.Fine-tuning/FineTuning_Vision.md

@nguyenbh Hi thank you so much for the finetune example! The example stated that the average tokens of DocVQA dataset is about 2XXX, I wonder if there is a long input would that cause OOM error? Does script have protection about that?

@eddtsoi there's no explicit code for handling OOM. Note that the 2k average #tokens include the image tokens. In some cases that doesn't require high resolution, it is possible to finetune with a lower --num_crops to reduce the sequence length. For full finetuning, we have tested on 4x 48GB and 8x 32GB GPUs. If using (q)lora, I believe a single consumer level 24GB GPU works.