Hugging Face's logo

Abe13
/
juni-Mistral-7B-OpenOrca

PEFT
Model card Files Community
Abe13 commited on
Commit
c50889b
•
1 Parent(s): 194a6b1

Model save

Browse files
Files changed (2) hide show
  1. README.md +141 -40
  2. adapter_model.bin +1 -1
README.md CHANGED
@@ -1,47 +1,148 @@
1
  ---
2
- library_name: peft
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  ## Training procedure
5
 
 
6
 
7
- The following `bitsandbytes` quantization config was used during training:
8
- - quant_method: bitsandbytes
9
- - load_in_8bit: False
10
- - load_in_4bit: True
11
- - llm_int8_threshold: 6.0
12
- - llm_int8_skip_modules: None
13
- - llm_int8_enable_fp32_cpu_offload: False
14
- - llm_int8_has_fp16_weight: False
15
- - bnb_4bit_quant_type: nf4
16
- - bnb_4bit_use_double_quant: False
17
- - bnb_4bit_compute_dtype: float16
18
-
19
- The following `bitsandbytes` quantization config was used during training:
20
- - quant_method: bitsandbytes
21
- - load_in_8bit: False
22
- - load_in_4bit: True
23
- - llm_int8_threshold: 6.0
24
- - llm_int8_skip_modules: None
25
- - llm_int8_enable_fp32_cpu_offload: False
26
- - llm_int8_has_fp16_weight: False
27
- - bnb_4bit_quant_type: nf4
28
- - bnb_4bit_use_double_quant: False
29
- - bnb_4bit_compute_dtype: float16
30
-
31
- The following `bitsandbytes` quantization config was used during training:
32
- - quant_method: bitsandbytes
33
- - load_in_8bit: False
34
- - load_in_4bit: True
35
- - llm_int8_threshold: 6.0
36
- - llm_int8_skip_modules: None
37
- - llm_int8_enable_fp32_cpu_offload: False
38
- - llm_int8_has_fp16_weight: False
39
- - bnb_4bit_quant_type: nf4
40
- - bnb_4bit_use_double_quant: False
41
- - bnb_4bit_compute_dtype: float16
42
- ### Framework versions
43
 
44
- - PEFT 0.6.0.dev0
45
- - PEFT 0.6.0.dev0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
- - PEFT 0.6.0.dev0
 
 
 
 
1
  ---
2
+ license: apache-2.0
3
+ base_model: Open-Orca/Mistral-7B-OpenOrca
4
+ tags:
5
+ - generated_from_trainer
6
+ model-index:
7
+ - name: juni-Mistral-7B-OpenOrca
8
+ results: []
9
  ---
10
+
11
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
12
+ should probably proofread and complete it, then remove this comment. -->
13
+
14
+ # juni-Mistral-7B-OpenOrca
15
+
16
+ This model is a fine-tuned version of [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca) on the None dataset.
17
+ It achieves the following results on the evaluation set:
18
+ - Loss: 3.0758
19
+
20
+ ## Model description
21
+
22
+ More information needed
23
+
24
+ ## Intended uses & limitations
25
+
26
+ More information needed
27
+
28
+ ## Training and evaluation data
29
+
30
+ More information needed
31
+
32
  ## Training procedure
33
 
34
+ ### Training hyperparameters
35
 
36
+ The following hyperparameters were used during training:
37
+ - learning_rate: 0.0001
38
+ - train_batch_size: 2
39
+ - eval_batch_size: 2
40
+ - seed: 42
41
+ - gradient_accumulation_steps: 8
42
+ - total_train_batch_size: 16
43
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
44
+ - lr_scheduler_type: linear
45
+ - num_epochs: 10
46
+
47
+ ### Training results
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
+ | Training Loss | Epoch | Step | Validation Loss |
50
+ |:-------------:|:-----:|:----:|:---------------:|
51
+ | 1.7594 | 0.11 | 1 | 3.4155 |
52
+ | 1.7761 | 0.22 | 2 | 3.3643 |
53
+ | 1.6344 | 0.32 | 3 | 3.3129 |
54
+ | 1.8145 | 0.43 | 4 | 3.2624 |
55
+ | 1.7308 | 0.54 | 5 | 3.2462 |
56
+ | 1.6688 | 0.65 | 6 | 3.2282 |
57
+ | 1.8082 | 0.76 | 7 | 3.2052 |
58
+ | 1.5884 | 0.86 | 8 | 3.1957 |
59
+ | 1.6247 | 0.97 | 9 | 3.1926 |
60
+ | 1.7539 | 1.08 | 10 | 3.1759 |
61
+ | 1.6578 | 1.19 | 11 | 3.1674 |
62
+ | 1.661 | 1.3 | 12 | 3.1829 |
63
+ | 1.5935 | 1.41 | 13 | 3.1785 |
64
+ | 1.5209 | 1.51 | 14 | 3.1687 |
65
+ | 1.6052 | 1.62 | 15 | 3.1504 |
66
+ | 1.495 | 1.73 | 16 | 3.1539 |
67
+ | 1.5238 | 1.84 | 17 | 3.1357 |
68
+ | 1.5698 | 1.95 | 18 | 3.1196 |
69
+ | 1.3628 | 2.05 | 19 | 3.1099 |
70
+ | 1.5966 | 2.16 | 20 | 3.1170 |
71
+ | 1.5713 | 2.27 | 21 | 3.1327 |
72
+ | 1.5321 | 2.38 | 22 | 3.1060 |
73
+ | 1.5511 | 2.49 | 23 | 3.1153 |
74
+ | 1.5605 | 2.59 | 24 | 3.0925 |
75
+ | 1.515 | 2.7 | 25 | 3.1066 |
76
+ | 1.4646 | 2.81 | 26 | 3.1005 |
77
+ | 1.3957 | 2.92 | 27 | 3.1305 |
78
+ | 1.4377 | 3.03 | 28 | 3.1143 |
79
+ | 1.4452 | 3.14 | 29 | 3.1472 |
80
+ | 1.4925 | 3.24 | 30 | 3.1050 |
81
+ | 1.4749 | 3.35 | 31 | 3.1264 |
82
+ | 1.5017 | 3.46 | 32 | 3.1107 |
83
+ | 1.5082 | 3.57 | 33 | 3.1000 |
84
+ | 1.4657 | 3.68 | 34 | 3.1220 |
85
+ | 1.2359 | 3.78 | 35 | 3.1199 |
86
+ | 1.4095 | 3.89 | 36 | 3.0966 |
87
+ | 1.5437 | 4.0 | 37 | 3.0847 |
88
+ | 1.339 | 4.11 | 38 | 3.1319 |
89
+ | 1.3762 | 4.22 | 39 | 3.0917 |
90
+ | 1.3964 | 4.32 | 40 | 3.0947 |
91
+ | 1.4472 | 4.43 | 41 | 3.1034 |
92
+ | 1.3863 | 4.54 | 42 | 3.1100 |
93
+ | 1.434 | 4.65 | 43 | 3.1018 |
94
+ | 1.5171 | 4.76 | 44 | 3.0831 |
95
+ | 1.215 | 4.86 | 45 | 3.0755 |
96
+ | 1.4791 | 4.97 | 46 | 3.0790 |
97
+ | 1.3341 | 5.08 | 47 | 3.0816 |
98
+ | 1.3899 | 5.19 | 48 | 3.0909 |
99
+ | 1.3621 | 5.3 | 49 | 3.0668 |
100
+ | 1.4034 | 5.41 | 50 | 3.0818 |
101
+ | 1.3541 | 5.51 | 51 | 3.0512 |
102
+ | 1.2916 | 5.62 | 52 | 3.0861 |
103
+ | 1.3359 | 5.73 | 53 | 3.0695 |
104
+ | 1.3962 | 5.84 | 54 | 3.0544 |
105
+ | 1.3537 | 5.95 | 55 | 3.0808 |
106
+ | 1.2551 | 6.05 | 56 | 3.0733 |
107
+ | 1.4321 | 6.16 | 57 | 3.0481 |
108
+ | 1.3511 | 6.27 | 58 | 3.0660 |
109
+ | 1.4584 | 6.38 | 59 | 3.0385 |
110
+ | 1.1897 | 6.49 | 60 | 3.0632 |
111
+ | 1.3157 | 6.59 | 61 | 3.0724 |
112
+ | 1.2269 | 6.7 | 62 | 3.0747 |
113
+ | 1.4017 | 6.81 | 63 | 3.0593 |
114
+ | 1.357 | 6.92 | 64 | 3.0655 |
115
+ | 1.4048 | 7.03 | 65 | 3.0649 |
116
+ | 1.308 | 7.14 | 66 | 3.0707 |
117
+ | 1.2297 | 7.24 | 67 | 3.0561 |
118
+ | 1.2186 | 7.35 | 68 | 3.0729 |
119
+ | 1.2583 | 7.46 | 69 | 3.0800 |
120
+ | 1.4283 | 7.57 | 70 | 3.0698 |
121
+ | 1.224 | 7.68 | 71 | 3.0787 |
122
+ | 1.2403 | 7.78 | 72 | 3.0669 |
123
+ | 1.2677 | 7.89 | 73 | 3.0615 |
124
+ | 1.3997 | 8.0 | 74 | 3.0658 |
125
+ | 1.2593 | 8.11 | 75 | 3.0714 |
126
+ | 1.1997 | 8.22 | 76 | 3.0752 |
127
+ | 1.2961 | 8.32 | 77 | 3.0662 |
128
+ | 1.3297 | 8.43 | 78 | 3.0637 |
129
+ | 1.2994 | 8.54 | 79 | 3.0660 |
130
+ | 1.3623 | 8.65 | 80 | 3.0626 |
131
+ | 1.1564 | 8.76 | 81 | 3.0658 |
132
+ | 1.3229 | 8.86 | 82 | 3.0674 |
133
+ | 1.1027 | 8.97 | 83 | 3.0688 |
134
+ | 1.3022 | 9.08 | 84 | 3.0699 |
135
+ | 1.2523 | 9.19 | 85 | 3.0684 |
136
+ | 1.198 | 9.3 | 86 | 3.0687 |
137
+ | 0.9721 | 9.41 | 87 | 3.0730 |
138
+ | 1.2124 | 9.51 | 88 | 3.0756 |
139
+ | 1.3073 | 9.62 | 89 | 3.0761 |
140
+ | 1.2945 | 9.73 | 90 | 3.0758 |
141
+
142
+
143
+ ### Framework versions
144
 
145
+ - Transformers 4.34.1
146
+ - Pytorch 2.0.1+cu118
147
+ - Datasets 2.14.6
148
+ - Tokenizers 0.14.1
adapter_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:29f0f2e048ef30dc20f64a441d36e43018cfd88e68a0affa72326d09b480a16a
3
  size 109097933
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:110cad8cfb7c0fba20936cd0e79c26474b0e3c2d3d58c63fc555939b1aaf14ca
3
  size 109097933