Added training config

README.md

@@ -61,7 +61,8 @@ print(tokenizer.decode(generation_output[0]))
 
 ### Training Data
 
-Converted the following datasets to alpaca:instruction format:
+Converted the following datasets to alpaca:instruction format.
+
 1. [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)
   - ORCA style dataset generously created by [Eric Hartford](https://huggingface.co/ehartford)
   - Only used the 1 million GPT4 generated instructions file [flan1m-alpaca-uncensored.jsonl](https://huggingface.co/datasets/ehartford/dolphin/blob/main/flan1m-alpaca-uncensored.jsonl).
@@ -76,9 +77,90 @@ Converted the following datasets to alpaca:instruction format:
 
 ### Training Procedure 
 
-Trained using axolotl QLoRa on RunPod 8x A6000 on Community Cloud for 2 epochs (~14 hours).
+Trained using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) QLoRa on [RunPod](https://www.runpod.io/console/gpu-cloud) 8x A6000 on Community Cloud for 3 epochs (~14 hours - ~$70).
 
-axolotl training config:
+<details>
+  <summary>axolotl training config:</summary>
+  
 ```yaml
+base_model: openlm-research/open_llama_7b_v2
+base_model_config: openlm-research/open_llama_7b_v2
+model_type: LlamaForCausalLM
+tokenizer_type: LlamaTokenizer
+load_in_8bit: false
+load_in_4bit: true
+strict: false
+
+push_dataset_to_hub:
+hub_model_id:
+hf_use_auth_token:
 
-```
+datasets:
+  - path: json
+    type: alpaca
+    data_files: /disk/flan1m-alpaca-uncensored.jsonl
+    shards: 8
+  - path: sahil2801/code_instructions_120k
+    type: alpaca
+  - path: LinhDuong/chatdoctor-200k
+    type: alpaca
+    shards: 2
+  - path: kaiokendev/SuperCOT-dataset
+    type: alpaca
+  - path: medalpaca/medical_meadow_mediqa
+    type: alpaca
+
+dataset_prepared_path: last_run_prepared
+val_set_size: 0.01
+adapter: qlora
+lora_model_dir:
+sequence_len: 2048
+max_packed_sequence_len: 2048
+lora_r: 8
+lora_alpha: 32
+lora_dropout: 0.05
+lora_target_modules:
+lora_target_linear: true
+lora_fan_in_fan_out:
+
+wandb_mode: true
+wandb_project:
+wandb_watch:
+wandb_run_id:
+wandb_log_model: 'openllama_checkpoint'
+output_dir: /disk/open_llama_7b_v2_dolphin_qlora
+gradient_accumulation_steps: 2
+micro_batch_size: 16
+num_epochs: 3
+optimizer: paged_adamw_32bit
+torchdistx_path:
+lr_scheduler: cosine
+learning_rate: 0.0002
+train_on_inputs: false
+group_by_length: false
+bf16: true
+fp16: false
+tf32: true
+gradient_checkpointing: true
+early_stopping_patience:
+resume_from_checkpoint:
+local_rank:
+logging_steps: 1
+xformers_attention: true
+flash_attention:
+gptq_groupsize:
+gptq_model_v1:
+warmup_steps: 1000
+eval_steps: 5000
+save_steps:
+debug:
+deepspeed:
+weight_decay: 0.0000001
+fsdp:
+fsdp_config:
+special_tokens:
+  bos_token: "<s>"
+  eos_token: "</s>"
+  unk_token: "<unk>"
+```
+</details>