File size: 821 Bytes
e70d9bf 175a00b e70d9bf b88c7b6 175a00b e09bb18 175a00b e09bb18 175a00b 6cb95b1 175a00b e09bb18 175a00b e09bb18 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
---
license: mit
datasets:
- yahma/alpaca-cleaned
---
This repo contains a low-rank adapter for LLaMA-13b fit on the Cleaned Alpaca dataset containing the new GPT-4 data.
This version of the weights was trained with the following hyperparameters:
Cleaned dataset: Snapshot April 9, 2023
Epochs: 4
Validation set size: 1500
Batch size: 128
Micro batch size: 4
Cutoff length: 512
Learning rate: 3e-4
Lora r: 16
Lora target modules: q_proj, k_proj, v_proj, o_proj
That is:
python finetune.py \
--base_model='yahma/llama-13b-hf' \
--data_path 'yahma/alpaca-cleaned' \
--num_epochs=4 \
--cutoff_len=512 \
--output_dir='./lora-alpaca' \
--lora_target_modules='[q_proj,k_proj, v_proj, o_proj]' \
--lora_r=16 \
--val_set_size 1500 \
--micro_batch_size=4 |