Hugging Face's logo

RemVdH
/
test-model-ft-tinylama

PEFT
Safetensors
llama
axolotl
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Community
test-model-ft-tinylama / README.md
RemVdH's picture
RemVdH
End of training
460819c verified
preview code
|
raw
history blame contribute delete
3.74 kB
metadata
license: apache-2.0
library_name: peft
tags:
  - axolotl
  - generated_from_trainer
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model-index:
  - name: test-model-ft-tinylama
    results: []

Built with Axolotl

See axolotl config

axolotl version: 0.4.0

base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: RemVdH/databricks-dolly-3k
    type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/lora-out

sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

hub_model_id: RemVdH/test-model-ft-tinylama

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

test-model-ft-tinylama

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.7487

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 10
  • num_epochs: 4

Training results

Training Loss Epoch Step Validation Loss
1.9375 0.0465 1 2.1187
1.9299 0.2791 6 2.0168
1.825 0.5581 12 1.8114
1.7291 0.8372 18 1.7892
1.7519 1.1047 24 1.7811
1.8679 1.3837 30 1.7753
1.6452 1.6628 36 1.7567
1.7842 1.9419 42 1.7574
1.6599 2.1977 48 1.7538
1.6158 2.4767 54 1.7543
1.7082 2.7558 60 1.7560
1.7263 3.0116 66 1.7518
1.8113 3.2907 72 1.7511
1.6883 3.5698 78 1.7497
1.7864 3.8488 84 1.7487

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.1.2+cu118
  • Datasets 2.19.1
  • Tokenizers 0.19.1