---
license: mit
datasets:
- yahma/alpaca-cleaned
---
This repo contains a low-rank adapter for LLaMA-13b fit on the Cleaned Alpaca dataset containing the new GPT-4 data.

This version of the weights was trained with the following hyperparameters:

    Cleaned dataset: Snapshot April 9, 2023
    Epochs: 4
    Validation set size: 1500
    Batch size: 128
    Micro batch size: 4
    Cutoff length: 512
    Learning rate: 3e-4
    Lora r: 16
    Lora target modules: q_proj, k_proj, v_proj, o_proj

That is:

python finetune.py \
    --base_model='decapoda-research/llama-13b-hf' \
    --data_path 'yahma/alpaca-cleaned' \
    --num_epochs=4 \
    --cutoff_len=512 \
    --output_dir='./lora-alpaca' \
    --lora_target_modules='[q_proj,k_proj, v_proj, o_proj]' \
    --lora_r=16 \
    --val_set_size 1500 \
    --micro_batch_size=4