|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- LumiOpen/instruction-collection-fin |
|
language: |
|
- fi |
|
- en |
|
--- |
|
<div align="center"> |
|
<img src="./poro-logo.png" width="200px"> |
|
</div> |
|
|
|
# Poro 34B Chat |
|
|
|
Poro 34b chat is a chat-tuned version of [Poro 34B](https://huggingface.co/LumiOpen/Poro-34B) trained to follow instructions in both Finnish and English. |
|
|
|
Because of the limited amount of instruction tuning available for Finnish, documents from the English datasets were machine-translated by the Poro 34B base model into Finnish, then used to train this chat version. We selected only datasets that are available for commercial use and only contain synthetic data if it was gathered in ToS-compliant fashion. |
|
|
|
More information about the data selection and translation process for our Finnish dataset are available on the [LumiOpen/instruction-collection-fin](https://huggingface.co/datasets/LumiOpen/instruction-collection-fin) page. |
|
|
|
Poro was created in a collaboration between [SiloGen](https://www.silo.ai/silogen) from [Silo AI](https://www.silo.ai/), the [TurkuNLP group](https://turkunlp.org/) of the University of Turku, and [High Performance Language Technologies](https://hplt-project.org/) (HPLT). Training was conducted on the [LUMI supercomputer](https://www.lumi-supercomputer.eu/), using compute resources generously provided by [CSC](https://csc.fi/) - IT Center for Science, Finland. |
|
|
|
This project is part of an ongoing effort to create open source large language models for non-English and especially low resource languages like Finnish. Through the combination of English and Finnish training data we get a model that outperforms previous Finnish only models, while also being fluent in English and code, and capable of basic translation between English and Finnish. |
|
|
|
|
|
## Fine Tuning |
|
Zephyr--??? TODO |
|
|
|
|
|
## Datasets |
|
|
|
#### Finnish and Cross-lingual |
|
- [LumiOpen/instruction-collection-fin](https://huggingface.co/datasets/LumiOpen/instruction-collection-fin) |
|
|
|
#### English |
|
- [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) |
|
- [Curated OASST2](https://huggingface.co/datasets/sablo/oasst2_curated) |
|
- [Argilla/10k_prompts_ranked_mistral_large_responses](https://huggingface.co/datasets/argilla/10k_prompts_ranked_mistral_large_responses) |
|
|
|
|
|
|
|
## Evaluations |
|
|
|
We relied on the popular MTBench benchmark to evaluate multi-turn performance. |
|
|
|
Since MTBench is an English only benchmark, we also release this fork of [MTBench Finnish](https://github.com/LumiOpen/FastChat/tree/main/fastchat/llm_judge) with multilingual support and machine translated Finnish prompts. Our scores for both benchmarks follow. |
|
|
|
| Eval | Score | |
|
| :-------------- | :----: | |
|
| MTBench | 5.93 | |
|
| MTBench Finnish | 5.90 | |
|
|
|
|
|
## License |
|
|
|
Poro 34B chat is released under the Apache 2.0 license. |
|
|
|
|
|
## Citation |
|
|
|
``` |
|
@misc{luukkonen2024poro, |
|
title={Poro 34B and the Blessing of Multilinguality}, |
|
author={Risto Luukkonen and Jonathan Burdge and Elaine Zosa and Aarne |
|
Talman and Ville Komulainen and Väinö Hatanpää and Peter Sarlin and Sampo |
|
Pyysalo}, |
|
year={2024}, |
|
eprint={2404.01856}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
