Hugging Face's logo

jtatman
/
pythia-125m-gpt4-llm-cleaned

PEFT
Safetensors
gpt_neox
axolotl
Generated from Trainer
Model card Files Community
pythia-125m-gpt4-llm-cleaned / README.md
jtatman's picture
jtatman
Training in progress, epoch 1
555e7c9 verified
preview code
|
raw
history blame
3.49 kB
metadata
base_model: EleutherAI/pythia-125m-deduped
library_name: peft
license: apache-2.0
tags:
  - axolotl
  - generated_from_trainer
model-index:
  - name: pythia-125m-gpt4-llm-cleaned
    results: []

Built with Axolotl

See axolotl config

axolotl version: 0.4.1

base_model: EleutherAI/pythia-125m-deduped
load_in_8bit: false
datasets:
  - path: teknium/GPT4-LLM-Cleaned
    type: alpaca
dataset_prepared_path: ds-gpt4-llm-cleaned
val_set_size: 0.05
adapter: lora
lora_model_dir:
sequence_len: 512
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - query_key_value
lora_target_linear: true 
lora_fan_in_fan_out: true  # pythia/GPTNeoX lora specific
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
output_dir: ./outputs/lora-alpaca-pythia-125m
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 4
learning_rate: 0.00001
train_on_inputs: false
group_by_length: false
bf16: auto
tf32: false
float16: true
gpu_memory_limit: 8GiB
hub_model_id: jtatman/pythia-125m-gpt4-llm-cleaned 
lora_on_cpu: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
weight_decay: 0.1
evals_per_epoch: 4
logging_steps: 1

pythia-125m-gpt4-llm-cleaned

This model is a fine-tuned version of EleutherAI/pythia-125m-deduped on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 2.0568

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 4

Training results

Training Loss Epoch Step Validation Loss
2.259 0.0001 1 3.3268
1.9508 0.25 3190 2.0963
2.0554 0.5 6380 2.0641
2.0165 0.75 9570 2.0588
1.9643 1.0 12760 2.0596
2.4041 1.25 15950 2.0576
1.7239 1.5 19140 2.0611
1.6508 1.75 22330 2.0557
1.8538 2.0 25520 2.0555
2.1191 2.25 28710 2.0586
2.4065 2.5 31900 2.0523
1.8175 2.75 35090 2.0534
2.3558 3.0 38280 2.0582
1.8991 3.25 41470 2.0544
2.1408 3.5 44660 2.0564
2.0138 3.75 47850 2.0568
2.0097 4.0 51040 2.0568

Framework versions

  • PEFT 0.11.1
  • Transformers 4.41.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1