multi task

#1
by vrashad - opened

Hello. How can we ask several questions in one prompt? For example: What is document name , total price , number of bank account and bank name ?
The model has only one answer to this question.

Owner

This is a great point! Because of the data the model was trained on, it makes sense that it won't generalize like that to many questions. What you can do is:

  1. Create datasets with the type of multiple questions that you want and fine-tune the model on them.
  2. Do some engineering with this model to split multiple questions and get multiple answers and re-stich them before presenting them to the user.

In any case, I would start from this model which was fine tuned for DocVQA:
https://huggingface.co/HuggingFaceM4/Florence-2-DocVQA

This is a great point! Because of the data the model was trained on, it makes sense that it won't generalize like that to many questions. What you can do is:

  1. Create datasets with the type of multiple questions that you want and fine-tune the model on them.
  2. Do some engineering with this model to split multiple questions and get multiple answers and re-stich them before presenting them to the user.

In any case, I would start from this model which was fine tuned for DocVQA:
https://huggingface.co/HuggingFaceM4/Florence-2-DocVQA

Thank you very much for your detailed answer.

andito changed discussion status to closed

Sign up or log in to comment