--- license: cc-by-nc-4.0 language: - en --- # Jellyfish-13B PicToModel ## Model Details Jellyfish-13B is a large language model with 13 billion parameters, designed specifically for data managment and preprocessing tasks, such as entity matching, data imputation, error detection, and schema matching. We fine-tuned [Open-Orca/OpenOrca-Platypus2-13B](https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B) using the datasets related to data preprocessing tasks. Its performance is competitive, standing up well against prior state-of-the-art algorithms and LLMs such as OpenAI GPT 3.5 and GPT 4 (evaluated by our previous work, https://arxiv.org/abs/2205.09911). Note that Jellyfish is only a 13B model and can be run locally for low cost and data security. | Task | Dataset | Non-LLM SoTA | GPT-3.5 | GPT-4 | Jellyfish-13B| Jellyfish-13B-Resoning | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | Entity Matching | Fodors-Zagats | 100 | 100 | 100 | 100 | 100 | | Entity Matching | Beer | 94.37| 96.30 | 100 | 93.33 | 100 | | Entity Matching | iTunes-Amazon | 97.06| 96.43 | 100 | 96.30 | 96.15 | | Entity Matching | Walmart-Amazon | 86.76| 86.17 | 90.27 | 80.71 | 85.16 | | Entity Matching | DBLP-ACM | 98.99| 96.99 | 97.44 | 97.35 | 95.74 | | Entity Matching | DBLP-GoogleScholar | 95.60| 76.12 | 91.87 | 92.83 | 89.45 | | Entity Matching | Amazon-Google | 75.58| 66.53 | 74.21 | 72.69 | 56.64 | | Imputation | Restaurant | 77.20| 94.19 | 97.67 | 94.19 | 93.02 | | Imputation | Buy | 96.50| 98.46 | 100 | 100 | 100 | | Error Detection | Hosptial | 99.10| 90.74 | 90.74 | 92.21 | 65.66 | | Error Detection | Adult | 94.40| 92.01 | 92.01 | 96.62 | 90.13 | | Schema Matching | Sythea | 38.50| 57.14 | 66.67 | 36.36 | 30.77 | We have released two versions of Jellyfish: the Jellyfish-13B and Jellyfish-13B-Reasoning. As the names suggest, Jellyfish-13B focuses on providing accurate, direct answers. In contrast, Jellyfish-13B-Reasoning distills knowledge from GPT-4. It fine-tuned with data containing reasons and chain-of-thought responses for solving data preprocessing tasks generated by GPT-4. **Jellyfish paper will coming soon!** - **Developed by:** Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada - **Contact: dongyuyang@nec.com** - **Funded by:** NEC Corporation, Osaka University - **Language(s) (NLP):** English - **License:** Non-Commercial Creative Commons license (CC BY-NC-4.0) - **Finetuned from model:** [Open-Orca/OpenOrca-Platypus2-13B](https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B) ## Prompt Template ``` ### Instruction: (without the <>) ### Response: ``` ## Training Details ### Training Data We utilized the training and validation sets from the paper [Can Foundation Models Wrangle Your Data?](https://arxiv.org/abs/2205.09911) to fine-tune Jellyfish The original datasets is [HazyResearch/fm_data_tasks](https://github.com/HazyResearch/fm_data_tasks). We revised this data and constructed an instruction tuning dataset suitable for fine-tuning LLM, mirroring the style of [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca). ### Training Method We used LoRA to speed up the training process, targeting the q_proj and v_proj modules. ## Uses Here are the prompts we used for both fine-tuning the model and for inference. Feel free to explore different prompts on your own to achieve the best generation quality. ### For JellyFish-13B ``` You are tasked with determining whether two records listed below are the same based on the information provided. Carefully compare the {attribute 1}, {attribute 2}... for each record before making your decision. Note: Missing values (N/A or \"nan\") should not be used as a basis for your decision. Record A: [{attribute 1}: {attribute 1 value}, {attribute 2}: {attribute 2 value}...]\nProduct B: [{attribute 1}: {attribute 1 value}, {attribute 2}: {attribute 2 value}...] Are record A and record B the same entity? Choose your answer from: [Yes, No] ``` ### For JellyFish-13B-reasoning ``` You are tasked with determining whether two products listed below are the same based on the information provided. Carefully examine all the attributes before making your decision. Note: Missing values (N/A or \"nan\") should not be used as a basis for your decision. Record A: [{attribute 1}: {attribute 1 value}, {attribute 2}: {attribute 2 value}...]\nProduct B: [{attribute 1}: {attribute 1 value}, {attribute 2}: {attribute 2 value}...] Are record A and record B the same entity? After your reasoning, finish your response in a separate line with and ONLY with your final answer. Choose your final answer from [Yes, No].", ``` ## Bias, Risks, and Limitations ## Citation