|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
library_name: transformers |
|
tags: |
|
- tenyx-fine-tuning |
|
- dpo |
|
- tenyxchat |
|
--- |
|
# TenyxChat: Language Model Alignment using Tenyx Fine-tuning |
|
|
|
Introducing TenyxChat-8x7B-v1, part of our TenyxChat series trained to function as useful assistants through preference tuning, using Tenyx's recently released advanced fine-tuning technology ([VentureBeat article](https://venturebeat.com/ai/tenyx-aims-to-fix-llms-catastrophic-forgetting-problem/)). Our model is trained using the [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290) framework on the open-source AI feedback dataset [UltraFeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized). |
|
|
|
We fine-tune [Mixtral-8x7B-Instruct-v0.1](https://arxiv.org/pdf/2401.04088.pdf) with our proprietary approach ([blog](https://www.tenyx.com/post/forgetting-and-toxicity-in-llms-a-deep-dive-on-fine-tuning-methods), [service](https://www.tenyx.com/fine-tuning)), |
|
similar to that of our [7B model](https://huggingface.co/tenyx/TenyxChat-7B-v1), and show an increase in [MT-Bench](https://arxiv.org/abs/2306.05685) scores. |
|
Our approach aims to mitigate forgetting in LLMs in a computationally efficient manner, thereby enabling continual fine-tuning capabilities without altering the pre-trained output distribution. |
|
TenyxChat-8x7B-v1 was trained using eight A100s (80GB) for about eight hours, with a training setup obtained from HuggingFaceH4 ([GitHub](https://github.com/huggingface/alignment-handbook)). |
|
|
|
# Model details |
|
|
|
- Model type: Fine-tuned Mixture Of Expert 8x7B model for chat. |
|
- License: Apache 2.0 |
|
- Base model: [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) |
|
- Demo: [spaces/tenyx/TenyxChat-8x7B-v1](https://huggingface.co/spaces/tenyx/TenyxChat-8x7B-v1) |
|
|
|
## Usage |
|
|
|
Our model uses a simple chat template based on Mixtral-8x7B-Instruct-v0.1 . The chat template usage with a Hugging face generation example is shown below. |
|
|
|
### Chat Template (Jinja) |
|
|
|
```rust |
|
{{ bos_token }} |
|
{% for message in messages %} |
|
{% if message['role'] == 'user' %} |
|
{{ '[INST]' + message['content'] + '[/INST]' }} |
|
{% elif message['role'] == 'system' %} |
|
{{ '[INST]' + message['content'] + '[/INST]' }} |
|
{% elif message['role'] == 'assistant' %} |
|
{{ message['content'] + eos_token }} |
|
{% endif %} |
|
{% endfor %} |
|
``` |
|
|
|
### Hugging face Example |
|
|
|
```python |
|
import torch |
|
from transformers import pipeline |
|
|
|
pipe = pipeline("text-generation", model="tenyx/TenyxChat-8x7B-v1", torch_dtype=torch.bfloat16, device_map="auto") |
|
|
|
messages = [ |
|
{"role": "system", "content": "You are a friendly chatbot who always responds in the style of a pirate."}, |
|
{"role": "user", "content": "Hi. I would like to make a hotel booking."}, |
|
] |
|
|
|
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
outputs = pipe(prompt, max_new_tokens=512, do_sample=False) |
|
``` |
|
|
|
|
|
# Performance |
|
|
|
At the time of release (Jan 2024), TenyxChat-8x7B-v1 is the highest-ranked model, only superseded by GPT4, on the MT-Bench evaluation available for download and commercial use. |
|
|
|
## MT-Bench |
|
|
|
MT-Bench is a benchmark made up of 80 high-quality multi-turn questions. These questions fall into eight categories: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, and Humanities. The chat models are rated using GPT-4 on a scale of 1 to 10, with higher values corresponding to better responses. |
|
|
|
|
|
| Model | First Turn | Second Turn | Average | |
|
| --- | --- | --- | --- | |
|
| GPT-4* | 8.95625 | 9.02500 | 8.990625 | |
|
| TenyxChat-8x7B-v1 | 8.63750 | 8.16250 | 8.400000 | |
|
| Mixtral (reproduced) | 8.49375 | 8.00000 | 8.246875 | |
|
| GPT-3.5-turbo* | 8.07500 | 7.81250 | 7.943750 | |
|
|
|
*values reported on [lmsys](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) ChatBot Arena |
|
|
|
 |
|
|
|
# Limitations |
|
|
|
TenyxChat-8x7B-v1, like other language models, has its own set of limitations. We haven’t fine-tuned the model explicitly to align with **human** safety preferences. Therefore, it is capable of producing undesirable outputs, particularly when adversarially prompted. From our observation, the model still tends to struggle with tasks that involve reasoning and math questions. In some instances, it might generate verbose or extraneous content. |
|
|
|
# License |
|
|
|
TenyxChat-8x7B-v1, similar to Mixtral-8x7B-Instruct-v0.1 , is distributed under the Apache License 2.0. |
|
|
|
# Citation |
|
|
|
If you use TenyxChat-8x7B-v1 for your research, cite us as |
|
|
|
``` |
|
@misc{tenyxchat2024, |
|
title={TenyxChat: Language Model Alignment using Tenyx Fine-tuning}, |
|
author={Tenyx}, |
|
year={2024}, |
|
} |
|
``` |
|
|
