---
library_name: transformers
license: llama3.1
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
tags:
- alignment-handbook
- trl
- cpo
- generated_from_trainer
- trl
- cpo
- generated_from_trainer
datasets:
- princeton-nlp/llama3-ultrafeedback
model-index:
- name: llama3.1-cpo-full-0912
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# llama3.1-cpo-full-0912

This model is a fine-tuned version of [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) on the princeton-nlp/llama3-ultrafeedback dataset.
It achieves the following results on the evaluation set:
- Loss: 1.5985
- Rewards/chosen: -15.4365
- Rewards/rejected: -16.1367
- Rewards/accuracies: 0.6239
- Rewards/margins: 0.7002
- Logps/rejected: -161.3668
- Logps/chosen: -154.3647
- Logits/rejected: -0.3853
- Logits/chosen: -0.4112
- Nll Loss: 0.4210

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|
| 1.9362        | 0.2311 | 100  | 1.7930          | -14.9339       | -15.2848         | 0.5761             | 0.3508          | -152.8475      | -149.3394    | -0.4123         | -0.4378       | 0.4067   |
| 1.7019        | 0.4623 | 200  | 1.6786          | -15.4303       | -16.0131         | 0.6087             | 0.5828          | -160.1311      | -154.3027    | -0.3358         | -0.3580       | 0.4193   |
| 1.6388        | 0.6934 | 300  | 1.6233          | -15.5465       | -16.2127         | 0.6130             | 0.6662          | -162.1269      | -155.4650    | -0.3582         | -0.3828       | 0.4230   |
| 1.632         | 0.9246 | 400  | 1.6007          | -15.6505       | -16.3448         | 0.6370             | 0.6943          | -163.4479      | -156.5048    | -0.3811         | -0.4072       | 0.4277   |


### Framework versions

- Transformers 4.44.2
- Pytorch 2.3.1
- Datasets 2.21.0
- Tokenizers 0.19.1