license: llama3
base_model: catallama/CataLlama-v0.1-Base
tags:
- llama
- llama-3
- Catalan
model-index:
- name: CataLlama-v0.1-Instruct-SFT
results: []
datasets:
- catallama/Catalan-Instruct
language:
- ca
- en
pipeline_tag: text-generation
CataLlama-v0.1-Instruct-SFT is an instruct fine-tune of catallama/CataLlama-v0.1-Base on the catallama/Catalan-Instruct dataset.
The model shows improved proficiency with the Catalan language.
This is an instruction fine-tuned model proficient on the following tasks in Catalan
- Information extraction (suitable for RAG)
- Named Entity Recognition (NER)
- Translation from English to Catalan and Catalan to English
- Summarization - both short form and long form
- Chat
- Sentiment analysis
- Open question answering
The model achieves a loss rate of 0.8528 on the validation dataset after two epochs.
Model developers Laurentiu Petrea based on Llama-3 from Meta.
Model Architecture CataLlama is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and direct preference optimisation (DPO) to align with human preferences for helpfulness and safety.
License The model uses the llama-3 license available at: https://llama.meta.com/llama3/license
Use with transformers
See the snippet below for usage with Transformers:
The model follows the same prompt template as Llama-3 Instruct
import transformers
import torch
model_id = "catallama/CataLlama-v0.1-Base"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)
messages = [
{"role": "user", "content": "Ei com estàs avui?"},
]
prompt = pipeline.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
outputs = pipeline(
prompt,
max_new_tokens=1024,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])
Training procedure
The model was trained with the same prompt template of Llama-3 Instruct.
The model was trained for two epochs on 6x A100 80GB GPUs using DeepSpeed ZeRO State-3 without CPU offloading.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- distributed_type: multi-GPU
- num_devices: 6
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 2
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.0186 | 0.22 | 200 | 1.0209 |
| 0.9588 | 0.43 | 400 | 0.9489 |
| 0.9111 | 0.65 | 600 | 0.9086 |
| 0.8971 | 0.86 | 800 | 0.8886 |
| 0.8002 | 1.22 | 1000 | 0.8989 |
| 0.8068 | 1.43 | 1200 | 0.8835 |
| 0.7722 | 1.65 | 1400 | 0.8654 |
| 0.7805 | 1.86 | 1600 | 0.8528 |