Hugging Face's logo

ondevicellm
/
tinyllama_moe_sft_ultrachat200k_v2

Text Generation
Transformers
TensorBoard
Safetensors
mixtral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
hushell commited on
Commit
a90fe41
•
1 Parent(s): 872baf8

Model save

Browse files
Files changed (12) hide show
  1. README.md +77 -0
  2. all_results.json +13 -0
  3. eval_results.json +8 -0
  4. generation_config.json +7 -0
  5. model-00001-of-00003.safetensors +3 -0
  6. model-00002-of-00003.safetensors +3 -0
  7. model-00003-of-00003.safetensors +3 -0
  8. model.safetensors.index.json +692 -0
  9. runs/Jan17_10-53-10_main1/events.out.tfevents.1705489747.main1.11008.0 +2 -2
  10. runs/Jan17_10-53-10_main1/events.out.tfevents.1705508081.main1.11008.1 +3 -0
  11. train_results.json +8 -0
  12. trainer_state.json +1492 -0
README.md ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: ondevicellm/tinyllama_moe
4
+ tags:
5
+ - trl
6
+ - sft
7
+ - generated_from_trainer
8
+ datasets:
9
+ - generator
10
+ model-index:
11
+ - name: tinyllama_moe_sft_ultrachat200k_v2
12
+ results: []
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+ # tinyllama_moe_sft_ultrachat200k_v2
19
+
20
+ This model is a fine-tuned version of [ondevicellm/tinyllama_moe](https://huggingface.co/ondevicellm/tinyllama_moe) on the generator dataset.
21
+ It achieves the following results on the evaluation set:
22
+ - Loss: 1.1593
23
+
24
+ ## Model description
25
+
26
+ More information needed
27
+
28
+ ## Intended uses & limitations
29
+
30
+ More information needed
31
+
32
+ ## Training and evaluation data
33
+
34
+ More information needed
35
+
36
+ ## Training procedure
37
+
38
+ ### Training hyperparameters
39
+
40
+ The following hyperparameters were used during training:
41
+ - learning_rate: 2e-05
42
+ - train_batch_size: 16
43
+ - eval_batch_size: 8
44
+ - seed: 42
45
+ - distributed_type: multi-GPU
46
+ - num_devices: 4
47
+ - gradient_accumulation_steps: 2
48
+ - total_train_batch_size: 128
49
+ - total_eval_batch_size: 32
50
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
51
+ - lr_scheduler_type: cosine
52
+ - lr_scheduler_warmup_ratio: 0.1
53
+ - num_epochs: 1
54
+
55
+ ### Training results
56
+
57
+ | Training Loss | Epoch | Step | Validation Loss |
58
+ |:-------------:|:-----:|:----:|:---------------:|
59
+ | 1.336 | 0.09 | 100 | 1.3140 |
60
+ | 1.2426 | 0.18 | 200 | 1.2376 |
61
+ | 1.2083 | 0.26 | 300 | 1.2100 |
62
+ | 1.1862 | 0.35 | 400 | 1.1934 |
63
+ | 1.1567 | 0.44 | 500 | 1.1820 |
64
+ | 1.1777 | 0.53 | 600 | 1.1737 |
65
+ | 1.1666 | 0.61 | 700 | 1.1677 |
66
+ | 1.1531 | 0.7 | 800 | 1.1636 |
67
+ | 1.1525 | 0.79 | 900 | 1.1610 |
68
+ | 1.1396 | 0.88 | 1000 | 1.1596 |
69
+ | 1.1681 | 0.96 | 1100 | 1.1593 |
70
+
71
+
72
+ ### Framework versions
73
+
74
+ - Transformers 4.36.2
75
+ - Pytorch 2.1.2+cu118
76
+ - Datasets 2.14.6
77
+ - Tokenizers 0.15.0
all_results.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_loss": 1.159261703491211,
4
+ "eval_runtime": 437.8092,
5
+ "eval_samples": 23110,
6
+ "eval_samples_per_second": 36.925,
7
+ "eval_steps_per_second": 1.156,
8
+ "train_loss": 1.2278979039839963,
9
+ "train_runtime": 17901.7657,
10
+ "train_samples": 207865,
11
+ "train_samples_per_second": 8.159,
12
+ "train_steps_per_second": 0.064
13
+ }
eval_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_loss": 1.159261703491211,
4
+ "eval_runtime": 437.8092,
5
+ "eval_samples": 23110,
6
+ "eval_samples_per_second": 36.925,
7
+ "eval_steps_per_second": 1.156
8
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.36.2",
6
+ "use_cache": false
7
+ }
model-00001-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db380538664d7680eb1a4827c6932727ddd677527badb7c5d268865d93fa1d79
3
+ size 4984275728
model-00002-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1737a6421f41fa229f54aaa3b7b819c459d21dc358f5d27b53140f145785ecc6
3
+ size 4991625072
model-00003-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9514d2a9328a04d5352e1db6fdf10f00a1701b6c36c646a94844d4f13970cdde
3
+ size 2882730416
model.safetensors.index.json ADDED
@@ -0,0 +1,692 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 12858544128
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00003-of-00003.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00003.safetensors",
8
+ "model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
9
+ "model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
10
+ "model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
11
+ "model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
12
+ "model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
13
+ "model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
14
+ "model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
15
+ "model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
16
+ "model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
17
+ "model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
18
+ "model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
19
+ "model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
20
+ "model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
21
+ "model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
22
+ "model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
23
+ "model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
24
+ "model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
25
+ "model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
26
+ "model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
27
+ "model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
28
+ "model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
29
+ "model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
30
+ "model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
31
+ "model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
32
+ "model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
33
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
34
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
35
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
36
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
37
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
38
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
39
+ "model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
40
+ "model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
41
+ "model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
42
+ "model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
43
+ "model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
44
+ "model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
45
+ "model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
46
+ "model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
47
+ "model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
48
+ "model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
49
+ "model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
50
+ "model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
51
+ "model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
52
+ "model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
53
+ "model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
54
+ "model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
55
+ "model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
56
+ "model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
57
+ "model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
58
+ "model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
59
+ "model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
60
+ "model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
61
+ "model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
62
+ "model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
63
+ "model.layers.1.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
64
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
65
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
66
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
67
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
68
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
69
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
70
+ "model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
71
+ "model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
72
+ "model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
73
+ "model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
74
+ "model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
75
+ "model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
76
+ "model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
77
+ "model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
78
+ "model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
79
+ "model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
80
+ "model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
81
+ "model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
82
+ "model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
83
+ "model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
84
+ "model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
85
+ "model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
86
+ "model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
87
+ "model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
88
+ "model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
89
+ "model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
90
+ "model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
91
+ "model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
92
+ "model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
93
+ "model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
94
+ "model.layers.10.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
95
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
96
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
97
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
98
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
99
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
100
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
101
+ "model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
102
+ "model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
103
+ "model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
104
+ "model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
105
+ "model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
106
+ "model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
107
+ "model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
108
+ "model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
109
+ "model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
110
+ "model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
111
+ "model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
112
+ "model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
113
+ "model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
114
+ "model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
115
+ "model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
116
+ "model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
117
+ "model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
118
+ "model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
119
+ "model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
120
+ "model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
121
+ "model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
122
+ "model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
123
+ "model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
124
+ "model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
125
+ "model.layers.11.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
126
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
127
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
128
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
129
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
130
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
131
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
132
+ "model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
133
+ "model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
134
+ "model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
135
+ "model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
136
+ "model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
137
+ "model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
138
+ "model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
139
+ "model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
140
+ "model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
141
+ "model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
142
+ "model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
143
+ "model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
144
+ "model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
145
+ "model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
146
+ "model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
147
+ "model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
148
+ "model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
149
+ "model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
150
+ "model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
151
+ "model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
152
+ "model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
153
+ "model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
154
+ "model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
155
+ "model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
156
+ "model.layers.12.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
157
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
158
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
159
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
160
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
161
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
162
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
163
+ "model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
164
+ "model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
165
+ "model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
166
+ "model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
167
+ "model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
168
+ "model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
169
+ "model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
170
+ "model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
171
+ "model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
172
+ "model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
173
+ "model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
174
+ "model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
175
+ "model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
176
+ "model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
177
+ "model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
178
+ "model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
179
+ "model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
180
+ "model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
181
+ "model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
182
+ "model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
183
+ "model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
184
+ "model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
185
+ "model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
186
+ "model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
187
+ "model.layers.13.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
188
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
189
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
190
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
191
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
192
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
193
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
194
+ "model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
195
+ "model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
196
+ "model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
197
+ "model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
198
+ "model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
199
+ "model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
200
+ "model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
201
+ "model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
202
+ "model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
203
+ "model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
204
+ "model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
205
+ "model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
206
+ "model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
207
+ "model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
208
+ "model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
209
+ "model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
210
+ "model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
211
+ "model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
212
+ "model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
213
+ "model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
214
+ "model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
215
+ "model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
216
+ "model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
217
+ "model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
218
+ "model.layers.14.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
219
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
220
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
221
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
222
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
223
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
224
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
225
+ "model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
226
+ "model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
227
+ "model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
228
+ "model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
229
+ "model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
230
+ "model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
231
+ "model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
232
+ "model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
233
+ "model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
234
+ "model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
235
+ "model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
236
+ "model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
237
+ "model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
238
+ "model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
239
+ "model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
240
+ "model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
241
+ "model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
242
+ "model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
243
+ "model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
244
+ "model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
245
+ "model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
246
+ "model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
247
+ "model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
248
+ "model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
249
+ "model.layers.15.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
250
+ "model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
251
+ "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
252
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
253
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
254
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
255
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
256
+ "model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
257
+ "model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
258
+ "model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
259
+ "model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
260
+ "model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
261
+ "model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
262
+ "model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
263
+ "model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
264
+ "model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
265
+ "model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
266
+ "model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
267
+ "model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
268
+ "model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
269
+ "model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
270
+ "model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
271
+ "model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
272
+ "model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
273
+ "model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
274
+ "model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
275
+ "model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
276
+ "model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
277
+ "model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
278
+ "model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
279
+ "model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
280
+ "model.layers.16.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
281
+ "model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
282
+ "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
283
+ "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
284
+ "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
285
+ "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
286
+ "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
287
+ "model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
288
+ "model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
289
+ "model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
290
+ "model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
291
+ "model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
292
+ "model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
293
+ "model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
294
+ "model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
295
+ "model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
296
+ "model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
297
+ "model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
298
+ "model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
299
+ "model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
300
+ "model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
301
+ "model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
302
+ "model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
303
+ "model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
304
+ "model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
305
+ "model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
306
+ "model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
307
+ "model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
308
+ "model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
309
+ "model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
310
+ "model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
311
+ "model.layers.17.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
312
+ "model.layers.17.input_layernorm.weight": "model-00003-of-00003.safetensors",
313
+ "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
314
+ "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
315
+ "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
316
+ "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
317
+ "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
318
+ "model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
319
+ "model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
320
+ "model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
321
+ "model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
322
+ "model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
323
+ "model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
324
+ "model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
325
+ "model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
326
+ "model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
327
+ "model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
328
+ "model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
329
+ "model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
330
+ "model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
331
+ "model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
332
+ "model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
333
+ "model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
334
+ "model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
335
+ "model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
336
+ "model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
337
+ "model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
338
+ "model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
339
+ "model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
340
+ "model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
341
+ "model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
342
+ "model.layers.18.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
343
+ "model.layers.18.input_layernorm.weight": "model-00003-of-00003.safetensors",
344
+ "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
345
+ "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
346
+ "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
347
+ "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
348
+ "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
349
+ "model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
350
+ "model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
351
+ "model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
352
+ "model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
353
+ "model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
354
+ "model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
355
+ "model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
356
+ "model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
357
+ "model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
358
+ "model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
359
+ "model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
360
+ "model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
361
+ "model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
362
+ "model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
363
+ "model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
364
+ "model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
365
+ "model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
366
+ "model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
367
+ "model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
368
+ "model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
369
+ "model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
370
+ "model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
371
+ "model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
372
+ "model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
373
+ "model.layers.19.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
374
+ "model.layers.19.input_layernorm.weight": "model-00003-of-00003.safetensors",
375
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
376
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
377
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
378
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
379
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
380
+ "model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
381
+ "model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
382
+ "model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
383
+ "model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
384
+ "model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
385
+ "model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
386
+ "model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
387
+ "model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
388
+ "model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
389
+ "model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
390
+ "model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
391
+ "model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
392
+ "model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
393
+ "model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
394
+ "model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
395
+ "model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
396
+ "model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
397
+ "model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
398
+ "model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
399
+ "model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
400
+ "model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
401
+ "model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
402
+ "model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
403
+ "model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
404
+ "model.layers.2.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
405
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
406
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
407
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
408
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
409
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
410
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
411
+ "model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
412
+ "model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
413
+ "model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
414
+ "model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
415
+ "model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
416
+ "model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
417
+ "model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
418
+ "model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
419
+ "model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
420
+ "model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
421
+ "model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
422
+ "model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
423
+ "model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
424
+ "model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
425
+ "model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
426
+ "model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
427
+ "model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
428
+ "model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
429
+ "model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
430
+ "model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
431
+ "model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
432
+ "model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
433
+ "model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
434
+ "model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
435
+ "model.layers.20.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
436
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00003.safetensors",
437
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
438
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
439
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
440
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
441
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
442
+ "model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
443
+ "model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
444
+ "model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
445
+ "model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
446
+ "model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
447
+ "model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
448
+ "model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
449
+ "model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
450
+ "model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
451
+ "model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
452
+ "model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
453
+ "model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
454
+ "model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
455
+ "model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
456
+ "model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
457
+ "model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
458
+ "model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
459
+ "model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
460
+ "model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
461
+ "model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
462
+ "model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
463
+ "model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
464
+ "model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
465
+ "model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
466
+ "model.layers.21.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
467
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00003.safetensors",
468
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
469
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
470
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
471
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
472
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
473
+ "model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
474
+ "model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
475
+ "model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
476
+ "model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
477
+ "model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
478
+ "model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
479
+ "model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
480
+ "model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
481
+ "model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
482
+ "model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
483
+ "model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
484
+ "model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
485
+ "model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
486
+ "model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
487
+ "model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
488
+ "model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
489
+ "model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
490
+ "model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
491
+ "model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
492
+ "model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
493
+ "model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
494
+ "model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
495
+ "model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
496
+ "model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
497
+ "model.layers.3.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
498
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
499
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
500
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
501
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
502
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
503
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
504
+ "model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
505
+ "model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
506
+ "model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
507
+ "model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
508
+ "model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
509
+ "model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
510
+ "model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
511
+ "model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
512
+ "model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
513
+ "model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
514
+ "model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
515
+ "model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
516
+ "model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
517
+ "model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
518
+ "model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
519
+ "model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
520
+ "model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
521
+ "model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
522
+ "model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
523
+ "model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
524
+ "model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
525
+ "model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
526
+ "model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
527
+ "model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
528
+ "model.layers.4.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
529
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
530
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
531
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
532
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
533
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
534
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
535
+ "model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
536
+ "model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
537
+ "model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
538
+ "model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
539
+ "model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
540
+ "model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
541
+ "model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
542
+ "model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
543
+ "model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
544
+ "model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
545
+ "model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
546
+ "model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
547
+ "model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
548
+ "model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
549
+ "model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
550
+ "model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
551
+ "model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
552
+ "model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
553
+ "model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
554
+ "model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
555
+ "model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
556
+ "model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
557
+ "model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
558
+ "model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
559
+ "model.layers.5.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
560
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
561
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
562
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
563
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
564
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
565
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
566
+ "model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
567
+ "model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
568
+ "model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
569
+ "model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
570
+ "model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
571
+ "model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
572
+ "model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
573
+ "model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
574
+ "model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
575
+ "model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
576
+ "model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
577
+ "model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
578
+ "model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
579
+ "model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
580
+ "model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
581
+ "model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
582
+ "model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
583
+ "model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
584
+ "model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
585
+ "model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
586
+ "model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
587
+ "model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
588
+ "model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
589
+ "model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
590
+ "model.layers.6.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
591
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
592
+ "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
593
+ "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
594
+ "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
595
+ "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
596
+ "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
597
+ "model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
598
+ "model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
599
+ "model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
600
+ "model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
601
+ "model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
602
+ "model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
603
+ "model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
604
+ "model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
605
+ "model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
606
+ "model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
607
+ "model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
608
+ "model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
609
+ "model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
610
+ "model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
611
+ "model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
612
+ "model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
613
+ "model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
614
+ "model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
615
+ "model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
616
+ "model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
617
+ "model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
618
+ "model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
619
+ "model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
620
+ "model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
621
+ "model.layers.7.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
622
+ "model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
623
+ "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
624
+ "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
625
+ "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
626
+ "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
627
+ "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
628
+ "model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
629
+ "model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
630
+ "model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
631
+ "model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
632
+ "model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
633
+ "model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
634
+ "model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
635
+ "model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
636
+ "model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
637
+ "model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
638
+ "model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
639
+ "model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
640
+ "model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
641
+ "model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
642
+ "model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
643
+ "model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
644
+ "model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
645
+ "model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
646
+ "model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
647
+ "model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
648
+ "model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
649
+ "model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
650
+ "model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
651
+ "model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
652
+ "model.layers.8.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
653
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00003.safetensors",
654
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
655
+ "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
656
+ "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
657
+ "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
658
+ "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
659
+ "model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
660
+ "model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
661
+ "model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
662
+ "model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
663
+ "model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
664
+ "model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
665
+ "model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
666
+ "model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
667
+ "model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
668
+ "model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
669
+ "model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
670
+ "model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
671
+ "model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
672
+ "model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
673
+ "model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
674
+ "model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
675
+ "model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
676
+ "model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
677
+ "model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
678
+ "model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
679
+ "model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
680
+ "model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
681
+ "model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
682
+ "model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
683
+ "model.layers.9.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
684
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00003.safetensors",
685
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
686
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
687
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
688
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
689
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
690
+ "model.norm.weight": "model-00003-of-00003.safetensors"
691
+ }
692
+ }
runs/Jan17_10-53-10_main1/events.out.tfevents.1705489747.main1.11008.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e4a082f0d54c2ac0701a722967b88711906cdb80c7dd081ed9d91aa9c26f39f6
3
- size 42298
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5c7c52c4a94817595bfb21f078490290a6914939ee571cdfda4f827e26200a09
3
+ size 43908
runs/Jan17_10-53-10_main1/events.out.tfevents.1705508081.main1.11008.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:374e7a1209cbf7ab339c2e4e4b050f2660d4824c8a48a457e9d92df13620ee8c
3
+ size 359
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 1.2278979039839963,
4
+ "train_runtime": 17901.7657,
5
+ "train_samples": 207865,
6
+ "train_samples_per_second": 8.159,
7
+ "train_steps_per_second": 0.064
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,1492 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 1.159261703491211,
3
+ "best_model_checkpoint": "data/tinyllama_moe_sft_ultrachat200k_v2/checkpoint-1100",
4
+ "epoch": 0.9995619798510732,
5
+ "eval_steps": 100,
6
+ "global_step": 1141,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0,
13
+ "learning_rate": 1.7391304347826088e-07,
14
+ "loss": 2.7477,
15
+ "step": 1
16
+ },
17
+ {
18
+ "epoch": 0.0,
19
+ "learning_rate": 8.695652173913044e-07,
20
+ "loss": 2.6989,
21
+ "step": 5
22
+ },
23
+ {
24
+ "epoch": 0.01,
25
+ "learning_rate": 1.7391304347826088e-06,
26
+ "loss": 2.7026,
27
+ "step": 10
28
+ },
29
+ {
30
+ "epoch": 0.01,
31
+ "learning_rate": 2.6086956521739132e-06,
32
+ "loss": 2.5994,
33
+ "step": 15
34
+ },
35
+ {
36
+ "epoch": 0.02,
37
+ "learning_rate": 3.4782608695652175e-06,
38
+ "loss": 2.4353,
39
+ "step": 20
40
+ },
41
+ {
42
+ "epoch": 0.02,
43
+ "learning_rate": 4.347826086956522e-06,
44
+ "loss": 2.0773,
45
+ "step": 25
46
+ },
47
+ {
48
+ "epoch": 0.03,
49
+ "learning_rate": 5.2173913043478265e-06,
50
+ "loss": 1.8706,
51
+ "step": 30
52
+ },
53
+ {
54
+ "epoch": 0.03,
55
+ "learning_rate": 6.086956521739132e-06,
56
+ "loss": 1.7352,
57
+ "step": 35
58
+ },
59
+ {
60
+ "epoch": 0.04,
61
+ "learning_rate": 6.956521739130435e-06,
62
+ "loss": 1.6574,
63
+ "step": 40
64
+ },
65
+ {
66
+ "epoch": 0.04,
67
+ "learning_rate": 7.82608695652174e-06,
68
+ "loss": 1.5879,
69
+ "step": 45
70
+ },
71
+ {
72
+ "epoch": 0.04,
73
+ "learning_rate": 8.695652173913044e-06,
74
+ "loss": 1.5429,
75
+ "step": 50
76
+ },
77
+ {
78
+ "epoch": 0.05,
79
+ "learning_rate": 9.565217391304349e-06,
80
+ "loss": 1.4804,
81
+ "step": 55
82
+ },
83
+ {
84
+ "epoch": 0.05,
85
+ "learning_rate": 1.0434782608695653e-05,
86
+ "loss": 1.4633,
87
+ "step": 60
88
+ },
89
+ {
90
+ "epoch": 0.06,
91
+ "learning_rate": 1.1304347826086957e-05,
92
+ "loss": 1.4199,
93
+ "step": 65
94
+ },
95
+ {
96
+ "epoch": 0.06,
97
+ "learning_rate": 1.2173913043478263e-05,
98
+ "loss": 1.4117,
99
+ "step": 70
100
+ },
101
+ {
102
+ "epoch": 0.07,
103
+ "learning_rate": 1.3043478260869566e-05,
104
+ "loss": 1.3667,
105
+ "step": 75
106
+ },
107
+ {
108
+ "epoch": 0.07,
109
+ "learning_rate": 1.391304347826087e-05,
110
+ "loss": 1.359,
111
+ "step": 80
112
+ },
113
+ {
114
+ "epoch": 0.07,
115
+ "learning_rate": 1.4782608695652174e-05,
116
+ "loss": 1.3563,
117
+ "step": 85
118
+ },
119
+ {
120
+ "epoch": 0.08,
121
+ "learning_rate": 1.565217391304348e-05,
122
+ "loss": 1.3425,
123
+ "step": 90
124
+ },
125
+ {
126
+ "epoch": 0.08,
127
+ "learning_rate": 1.6521739130434785e-05,
128
+ "loss": 1.3117,
129
+ "step": 95
130
+ },
131
+ {
132
+ "epoch": 0.09,
133
+ "learning_rate": 1.739130434782609e-05,
134
+ "loss": 1.336,
135
+ "step": 100
136
+ },
137
+ {
138
+ "epoch": 0.09,
139
+ "eval_loss": 1.3140063285827637,
140
+ "eval_runtime": 441.4184,
141
+ "eval_samples_per_second": 36.623,
142
+ "eval_steps_per_second": 1.146,
143
+ "step": 100
144
+ },
145
+ {
146
+ "epoch": 0.09,
147
+ "learning_rate": 1.8260869565217393e-05,
148
+ "loss": 1.3087,
149
+ "step": 105
150
+ },
151
+ {
152
+ "epoch": 0.1,
153
+ "learning_rate": 1.9130434782608697e-05,
154
+ "loss": 1.2993,
155
+ "step": 110
156
+ },
157
+ {
158
+ "epoch": 0.1,
159
+ "learning_rate": 2e-05,
160
+ "loss": 1.2998,
161
+ "step": 115
162
+ },
163
+ {
164
+ "epoch": 0.11,
165
+ "learning_rate": 1.99988280568259e-05,
166
+ "loss": 1.2738,
167
+ "step": 120
168
+ },
169
+ {
170
+ "epoch": 0.11,
171
+ "learning_rate": 1.9995312501993765e-05,
172
+ "loss": 1.277,
173
+ "step": 125
174
+ },
175
+ {
176
+ "epoch": 0.11,
177
+ "learning_rate": 1.998945415950969e-05,
178
+ "loss": 1.255,
179
+ "step": 130
180
+ },
181
+ {
182
+ "epoch": 0.12,
183
+ "learning_rate": 1.9981254402502568e-05,
184
+ "loss": 1.271,
185
+ "step": 135
186
+ },
187
+ {
188
+ "epoch": 0.12,
189
+ "learning_rate": 1.9970715152902257e-05,
190
+ "loss": 1.282,
191
+ "step": 140
192
+ },
193
+ {
194
+ "epoch": 0.13,
195
+ "learning_rate": 1.9957838880989076e-05,
196
+ "loss": 1.2598,
197
+ "step": 145
198
+ },
199
+ {
200
+ "epoch": 0.13,
201
+ "learning_rate": 1.9942628604814827e-05,
202
+ "loss": 1.2623,
203
+ "step": 150
204
+ },
205
+ {
206
+ "epoch": 0.14,
207
+ "learning_rate": 1.9925087889495374e-05,
208
+ "loss": 1.2542,
209
+ "step": 155
210
+ },
211
+ {
212
+ "epoch": 0.14,
213
+ "learning_rate": 1.990522084637503e-05,
214
+ "loss": 1.2481,
215
+ "step": 160
216
+ },
217
+ {
218
+ "epoch": 0.14,
219
+ "learning_rate": 1.9883032132062926e-05,
220
+ "loss": 1.2445,
221
+ "step": 165
222
+ },
223
+ {
224
+ "epoch": 0.15,
225
+ "learning_rate": 1.98585269473415e-05,
226
+ "loss": 1.2483,
227
+ "step": 170
228
+ },
229
+ {
230
+ "epoch": 0.15,
231
+ "learning_rate": 1.9831711035947552e-05,
232
+ "loss": 1.2575,
233
+ "step": 175
234
+ },
235
+ {
236
+ "epoch": 0.16,
237
+ "learning_rate": 1.9802590683225945e-05,
238
+ "loss": 1.2308,
239
+ "step": 180
240
+ },
241
+ {
242
+ "epoch": 0.16,
243
+ "learning_rate": 1.97711727146564e-05,
244
+ "loss": 1.2328,
245
+ "step": 185
246
+ },
247
+ {
248
+ "epoch": 0.17,
249
+ "learning_rate": 1.973746449425368e-05,
250
+ "loss": 1.2423,
251
+ "step": 190
252
+ },
253
+ {
254
+ "epoch": 0.17,
255
+ "learning_rate": 1.970147392284154e-05,
256
+ "loss": 1.2272,
257
+ "step": 195
258
+ },
259
+ {
260
+ "epoch": 0.18,
261
+ "learning_rate": 1.9663209436200887e-05,
262
+ "loss": 1.2426,
263
+ "step": 200
264
+ },
265
+ {
266
+ "epoch": 0.18,
267
+ "eval_loss": 1.2375621795654297,
268
+ "eval_runtime": 439.5821,
269
+ "eval_samples_per_second": 36.776,
270
+ "eval_steps_per_second": 1.151,
271
+ "step": 200
272
+ },
273
+ {
274
+ "epoch": 0.18,
275
+ "learning_rate": 1.9622680003092503e-05,
276
+ "loss": 1.2404,
277
+ "step": 205
278
+ },
279
+ {
280
+ "epoch": 0.18,
281
+ "learning_rate": 1.957989512315489e-05,
282
+ "loss": 1.2137,
283
+ "step": 210
284
+ },
285
+ {
286
+ "epoch": 0.19,
287
+ "learning_rate": 1.953486482467764e-05,
288
+ "loss": 1.2462,
289
+ "step": 215
290
+ },
291
+ {
292
+ "epoch": 0.19,
293
+ "learning_rate": 1.9487599662250945e-05,
294
+ "loss": 1.2199,
295
+ "step": 220
296
+ },
297
+ {
298
+ "epoch": 0.2,
299
+ "learning_rate": 1.9438110714291697e-05,
300
+ "loss": 1.2038,
301
+ "step": 225
302
+ },
303
+ {
304
+ "epoch": 0.2,
305
+ "learning_rate": 1.9386409580446846e-05,
306
+ "loss": 1.2223,
307
+ "step": 230
308
+ },
309
+ {
310
+ "epoch": 0.21,
311
+ "learning_rate": 1.933250837887457e-05,
312
+ "loss": 1.2325,
313
+ "step": 235
314
+ },
315
+ {
316
+ "epoch": 0.21,
317
+ "learning_rate": 1.9276419743403934e-05,
318
+ "loss": 1.2049,
319
+ "step": 240
320
+ },
321
+ {
322
+ "epoch": 0.21,
323
+ "learning_rate": 1.9218156820573618e-05,
324
+ "loss": 1.2341,
325
+ "step": 245
326
+ },
327
+ {
328
+ "epoch": 0.22,
329
+ "learning_rate": 1.9157733266550577e-05,
330
+ "loss": 1.2086,
331
+ "step": 250
332
+ },
333
+ {
334
+ "epoch": 0.22,
335
+ "learning_rate": 1.9095163243929143e-05,
336
+ "loss": 1.2172,
337
+ "step": 255
338
+ },
339
+ {
340
+ "epoch": 0.23,
341
+ "learning_rate": 1.9030461418411498e-05,
342
+ "loss": 1.2212,
343
+ "step": 260
344
+ },
345
+ {
346
+ "epoch": 0.23,
347
+ "learning_rate": 1.8963642955370203e-05,
348
+ "loss": 1.2019,
349
+ "step": 265
350
+ },
351
+ {
352
+ "epoch": 0.24,
353
+ "learning_rate": 1.889472351629358e-05,
354
+ "loss": 1.2031,
355
+ "step": 270
356
+ },
357
+ {
358
+ "epoch": 0.24,
359
+ "learning_rate": 1.882371925511488e-05,
360
+ "loss": 1.2174,
361
+ "step": 275
362
+ },
363
+ {
364
+ "epoch": 0.25,
365
+ "learning_rate": 1.875064681442594e-05,
366
+ "loss": 1.1925,
367
+ "step": 280
368
+ },
369
+ {
370
+ "epoch": 0.25,
371
+ "learning_rate": 1.867552332157637e-05,
372
+ "loss": 1.2244,
373
+ "step": 285
374
+ },
375
+ {
376
+ "epoch": 0.25,
377
+ "learning_rate": 1.8598366384659113e-05,
378
+ "loss": 1.1928,
379
+ "step": 290
380
+ },
381
+ {
382
+ "epoch": 0.26,
383
+ "learning_rate": 1.851919408838327e-05,
384
+ "loss": 1.2125,
385
+ "step": 295
386
+ },
387
+ {
388
+ "epoch": 0.26,
389
+ "learning_rate": 1.843802498983529e-05,
390
+ "loss": 1.2083,
391
+ "step": 300
392
+ },
393
+ {
394
+ "epoch": 0.26,
395
+ "eval_loss": 1.2099547386169434,
396
+ "eval_runtime": 440.0729,
397
+ "eval_samples_per_second": 36.735,
398
+ "eval_steps_per_second": 1.15,
399
+ "step": 300
400
+ },
401
+ {
402
+ "epoch": 0.27,
403
+ "learning_rate": 1.8354878114129368e-05,
404
+ "loss": 1.2108,
405
+ "step": 305
406
+ },
407
+ {
408
+ "epoch": 0.27,
409
+ "learning_rate": 1.8269772949948185e-05,
410
+ "loss": 1.1779,
411
+ "step": 310
412
+ },
413
+ {
414
+ "epoch": 0.28,
415
+ "learning_rate": 1.8182729444974993e-05,
416
+ "loss": 1.2009,
417
+ "step": 315
418
+ },
419
+ {
420
+ "epoch": 0.28,
421
+ "learning_rate": 1.8093768001218096e-05,
422
+ "loss": 1.1952,
423
+ "step": 320
424
+ },
425
+ {
426
+ "epoch": 0.28,
427
+ "learning_rate": 1.800290947022884e-05,
428
+ "loss": 1.1917,
429
+ "step": 325
430
+ },
431
+ {
432
+ "epoch": 0.29,
433
+ "learning_rate": 1.7910175148214274e-05,
434
+ "loss": 1.1865,
435
+ "step": 330
436
+ },
437
+ {
438
+ "epoch": 0.29,
439
+ "learning_rate": 1.7815586771045535e-05,
440
+ "loss": 1.238,
441
+ "step": 335
442
+ },
443
+ {
444
+ "epoch": 0.3,
445
+ "learning_rate": 1.771916650916321e-05,
446
+ "loss": 1.1902,
447
+ "step": 340
448
+ },
449
+ {
450
+ "epoch": 0.3,
451
+ "learning_rate": 1.762093696238086e-05,
452
+ "loss": 1.1978,
453
+ "step": 345
454
+ },
455
+ {
456
+ "epoch": 0.31,
457
+ "learning_rate": 1.752092115458784e-05,
458
+ "loss": 1.1917,
459
+ "step": 350
460
+ },
461
+ {
462
+ "epoch": 0.31,
463
+ "learning_rate": 1.7419142528352815e-05,
464
+ "loss": 1.1908,
465
+ "step": 355
466
+ },
467
+ {
468
+ "epoch": 0.32,
469
+ "learning_rate": 1.731562493942904e-05,
470
+ "loss": 1.1964,
471
+ "step": 360
472
+ },
473
+ {
474
+ "epoch": 0.32,
475
+ "learning_rate": 1.721039265116285e-05,
476
+ "loss": 1.1929,
477
+ "step": 365
478
+ },
479
+ {
480
+ "epoch": 0.32,
481
+ "learning_rate": 1.710347032880664e-05,
482
+ "loss": 1.1871,
483
+ "step": 370
484
+ },
485
+ {
486
+ "epoch": 0.33,
487
+ "learning_rate": 1.6994883033737582e-05,
488
+ "loss": 1.1787,
489
+ "step": 375
490
+ },
491
+ {
492
+ "epoch": 0.33,
493
+ "learning_rate": 1.688465621758352e-05,
494
+ "loss": 1.1942,
495
+ "step": 380
496
+ },
497
+ {
498
+ "epoch": 0.34,
499
+ "learning_rate": 1.6772815716257414e-05,
500
+ "loss": 1.1841,
501
+ "step": 385
502
+ },
503
+ {
504
+ "epoch": 0.34,
505
+ "learning_rate": 1.6659387743901688e-05,
506
+ "loss": 1.1773,
507
+ "step": 390
508
+ },
509
+ {
510
+ "epoch": 0.35,
511
+ "learning_rate": 1.6544398886743934e-05,
512
+ "loss": 1.1789,
513
+ "step": 395
514
+ },
515
+ {
516
+ "epoch": 0.35,
517
+ "learning_rate": 1.6427876096865394e-05,
518
+ "loss": 1.1862,
519
+ "step": 400
520
+ },
521
+ {
522
+ "epoch": 0.35,
523
+ "eval_loss": 1.1934226751327515,
524
+ "eval_runtime": 441.3165,
525
+ "eval_samples_per_second": 36.631,
526
+ "eval_steps_per_second": 1.147,
527
+ "step": 400
528
+ },
529
+ {
530
+ "epoch": 0.35,
531
+ "learning_rate": 1.6309846685883726e-05,
532
+ "loss": 1.2072,
533
+ "step": 405
534
+ },
535
+ {
536
+ "epoch": 0.36,
537
+ "learning_rate": 1.6190338318551426e-05,
538
+ "loss": 1.1989,
539
+ "step": 410
540
+ },
541
+ {
542
+ "epoch": 0.36,
543
+ "learning_rate": 1.606937900627157e-05,
544
+ "loss": 1.1853,
545
+ "step": 415
546
+ },
547
+ {
548
+ "epoch": 0.37,
549
+ "learning_rate": 1.594699710053223e-05,
550
+ "loss": 1.1869,
551
+ "step": 420
552
+ },
553
+ {
554
+ "epoch": 0.37,
555
+ "learning_rate": 1.5823221286261217e-05,
556
+ "loss": 1.1782,
557
+ "step": 425
558
+ },
559
+ {
560
+ "epoch": 0.38,
561
+ "learning_rate": 1.5698080575102662e-05,
562
+ "loss": 1.1856,
563
+ "step": 430
564
+ },
565
+ {
566
+ "epoch": 0.38,
567
+ "learning_rate": 1.557160429861702e-05,
568
+ "loss": 1.1724,
569
+ "step": 435
570
+ },
571
+ {
572
+ "epoch": 0.39,
573
+ "learning_rate": 1.5443822101406066e-05,
574
+ "loss": 1.1868,
575
+ "step": 440
576
+ },
577
+ {
578
+ "epoch": 0.39,
579
+ "learning_rate": 1.531476393416456e-05,
580
+ "loss": 1.1764,
581
+ "step": 445
582
+ },
583
+ {
584
+ "epoch": 0.39,
585
+ "learning_rate": 1.5184460046660139e-05,
586
+ "loss": 1.2012,
587
+ "step": 450
588
+ },
589
+ {
590
+ "epoch": 0.4,
591
+ "learning_rate": 1.50529409806431e-05,
592
+ "loss": 1.1718,
593
+ "step": 455
594
+ },
595
+ {
596
+ "epoch": 0.4,
597
+ "learning_rate": 1.4920237562687784e-05,
598
+ "loss": 1.1708,
599
+ "step": 460
600
+ },
601
+ {
602
+ "epoch": 0.41,
603
+ "learning_rate": 1.478638089696716e-05,
604
+ "loss": 1.1811,
605
+ "step": 465
606
+ },
607
+ {
608
+ "epoch": 0.41,
609
+ "learning_rate": 1.4651402357962368e-05,
610
+ "loss": 1.1852,
611
+ "step": 470
612
+ },
613
+ {
614
+ "epoch": 0.42,
615
+ "learning_rate": 1.4515333583108896e-05,
616
+ "loss": 1.1792,
617
+ "step": 475
618
+ },
619
+ {
620
+ "epoch": 0.42,
621
+ "learning_rate": 1.4378206465381122e-05,
622
+ "loss": 1.1866,
623
+ "step": 480
624
+ },
625
+ {
626
+ "epoch": 0.42,
627
+ "learning_rate": 1.4240053145816968e-05,
628
+ "loss": 1.1529,
629
+ "step": 485
630
+ },
631
+ {
632
+ "epoch": 0.43,
633
+ "learning_rate": 1.4100906005984404e-05,
634
+ "loss": 1.1858,
635
+ "step": 490
636
+ },
637
+ {
638
+ "epoch": 0.43,
639
+ "learning_rate": 1.396079766039157e-05,
640
+ "loss": 1.1694,
641
+ "step": 495
642
+ },
643
+ {
644
+ "epoch": 0.44,
645
+ "learning_rate": 1.381976094884232e-05,
646
+ "loss": 1.1567,
647
+ "step": 500
648
+ },
649
+ {
650
+ "epoch": 0.44,
651
+ "eval_loss": 1.181981086730957,
652
+ "eval_runtime": 440.5529,
653
+ "eval_samples_per_second": 36.695,
654
+ "eval_steps_per_second": 1.149,
655
+ "step": 500
656
+ },
657
+ {
658
+ "epoch": 0.44,
659
+ "learning_rate": 1.3677828928738934e-05,
660
+ "loss": 1.1706,
661
+ "step": 505
662
+ },
663
+ {
664
+ "epoch": 0.45,
665
+ "learning_rate": 1.3535034867333838e-05,
666
+ "loss": 1.1738,
667
+ "step": 510
668
+ },
669
+ {
670
+ "epoch": 0.45,
671
+ "learning_rate": 1.3391412233932148e-05,
672
+ "loss": 1.1816,
673
+ "step": 515
674
+ },
675
+ {
676
+ "epoch": 0.46,
677
+ "learning_rate": 1.3246994692046837e-05,
678
+ "loss": 1.1519,
679
+ "step": 520
680
+ },
681
+ {
682
+ "epoch": 0.46,
683
+ "learning_rate": 1.3101816091508389e-05,
684
+ "loss": 1.1715,
685
+ "step": 525
686
+ },
687
+ {
688
+ "epoch": 0.46,
689
+ "learning_rate": 1.2955910460530787e-05,
690
+ "loss": 1.162,
691
+ "step": 530
692
+ },
693
+ {
694
+ "epoch": 0.47,
695
+ "learning_rate": 1.2809311997735697e-05,
696
+ "loss": 1.1583,
697
+ "step": 535
698
+ },
699
+ {
700
+ "epoch": 0.47,
701
+ "learning_rate": 1.266205506413667e-05,
702
+ "loss": 1.173,
703
+ "step": 540
704
+ },
705
+ {
706
+ "epoch": 0.48,
707
+ "learning_rate": 1.2514174175085346e-05,
708
+ "loss": 1.1622,
709
+ "step": 545
710
+ },
711
+ {
712
+ "epoch": 0.48,
713
+ "learning_rate": 1.2365703992181425e-05,
714
+ "loss": 1.1742,
715
+ "step": 550
716
+ },
717
+ {
718
+ "epoch": 0.49,
719
+ "learning_rate": 1.2216679315148388e-05,
720
+ "loss": 1.1752,
721
+ "step": 555
722
+ },
723
+ {
724
+ "epoch": 0.49,
725
+ "learning_rate": 1.2067135073676841e-05,
726
+ "loss": 1.1819,
727
+ "step": 560
728
+ },
729
+ {
730
+ "epoch": 0.49,
731
+ "learning_rate": 1.1917106319237386e-05,
732
+ "loss": 1.179,
733
+ "step": 565
734
+ },
735
+ {
736
+ "epoch": 0.5,
737
+ "learning_rate": 1.1766628216864961e-05,
738
+ "loss": 1.1706,
739
+ "step": 570
740
+ },
741
+ {
742
+ "epoch": 0.5,
743
+ "learning_rate": 1.161573603691655e-05,
744
+ "loss": 1.1717,
745
+ "step": 575
746
+ },
747
+ {
748
+ "epoch": 0.51,
749
+ "learning_rate": 1.1464465146804218e-05,
750
+ "loss": 1.1563,
751
+ "step": 580
752
+ },
753
+ {
754
+ "epoch": 0.51,
755
+ "learning_rate": 1.1312851002705383e-05,
756
+ "loss": 1.1721,
757
+ "step": 585
758
+ },
759
+ {
760
+ "epoch": 0.52,
761
+ "learning_rate": 1.1160929141252303e-05,
762
+ "loss": 1.1635,
763
+ "step": 590
764
+ },
765
+ {
766
+ "epoch": 0.52,
767
+ "learning_rate": 1.1008735171202685e-05,
768
+ "loss": 1.1883,
769
+ "step": 595
770
+ },
771
+ {
772
+ "epoch": 0.53,
773
+ "learning_rate": 1.0856304765093391e-05,
774
+ "loss": 1.1777,
775
+ "step": 600
776
+ },
777
+ {
778
+ "epoch": 0.53,
779
+ "eval_loss": 1.1737236976623535,
780
+ "eval_runtime": 439.8017,
781
+ "eval_samples_per_second": 36.757,
782
+ "eval_steps_per_second": 1.151,
783
+ "step": 600
784
+ },
785
+ {
786
+ "epoch": 0.53,
787
+ "learning_rate": 1.0703673650879219e-05,
788
+ "loss": 1.1483,
789
+ "step": 605
790
+ },
791
+ {
792
+ "epoch": 0.53,
793
+ "learning_rate": 1.0550877603558656e-05,
794
+ "loss": 1.1606,
795
+ "step": 610
796
+ },
797
+ {
798
+ "epoch": 0.54,
799
+ "learning_rate": 1.0397952436788643e-05,
800
+ "loss": 1.1813,
801
+ "step": 615
802
+ },
803
+ {
804
+ "epoch": 0.54,
805
+ "learning_rate": 1.024493399449025e-05,
806
+ "loss": 1.1574,
807
+ "step": 620
808
+ },
809
+ {
810
+ "epoch": 0.55,
811
+ "learning_rate": 1.0091858142447266e-05,
812
+ "loss": 1.1614,
813
+ "step": 625
814
+ },
815
+ {
816
+ "epoch": 0.55,
817
+ "learning_rate": 9.938760759899674e-06,
818
+ "loss": 1.1617,
819
+ "step": 630
820
+ },
821
+ {
822
+ "epoch": 0.56,
823
+ "learning_rate": 9.785677731133972e-06,
824
+ "loss": 1.1814,
825
+ "step": 635
826
+ },
827
+ {
828
+ "epoch": 0.56,
829
+ "learning_rate": 9.632644937072277e-06,
830
+ "loss": 1.1725,
831
+ "step": 640
832
+ },
833
+ {
834
+ "epoch": 0.57,
835
+ "learning_rate": 9.479698246862277e-06,
836
+ "loss": 1.1692,
837
+ "step": 645
838
+ },
839
+ {
840
+ "epoch": 0.57,
841
+ "learning_rate": 9.326873509469887e-06,
842
+ "loss": 1.1591,
843
+ "step": 650
844
+ },
845
+ {
846
+ "epoch": 0.57,
847
+ "learning_rate": 9.174206545276678e-06,
848
+ "loss": 1.1652,
849
+ "step": 655
850
+ },
851
+ {
852
+ "epoch": 0.58,
853
+ "learning_rate": 9.021733137683963e-06,
854
+ "loss": 1.1701,
855
+ "step": 660
856
+ },
857
+ {
858
+ "epoch": 0.58,
859
+ "learning_rate": 8.869489024725595e-06,
860
+ "loss": 1.1608,
861
+ "step": 665
862
+ },
863
+ {
864
+ "epoch": 0.59,
865
+ "learning_rate": 8.717509890691369e-06,
866
+ "loss": 1.1485,
867
+ "step": 670
868
+ },
869
+ {
870
+ "epoch": 0.59,
871
+ "learning_rate": 8.565831357763039e-06,
872
+ "loss": 1.1717,
873
+ "step": 675
874
+ },
875
+ {
876
+ "epoch": 0.6,
877
+ "learning_rate": 8.414488977664858e-06,
878
+ "loss": 1.1788,
879
+ "step": 680
880
+ },
881
+ {
882
+ "epoch": 0.6,
883
+ "learning_rate": 8.263518223330698e-06,
884
+ "loss": 1.1595,
885
+ "step": 685
886
+ },
887
+ {
888
+ "epoch": 0.6,
889
+ "learning_rate": 8.112954480589558e-06,
890
+ "loss": 1.1612,
891
+ "step": 690
892
+ },
893
+ {
894
+ "epoch": 0.61,
895
+ "learning_rate": 7.962833039871562e-06,
896
+ "loss": 1.1542,
897
+ "step": 695
898
+ },
899
+ {
900
+ "epoch": 0.61,
901
+ "learning_rate": 7.813189087936243e-06,
902
+ "loss": 1.1666,
903
+ "step": 700
904
+ },
905
+ {
906
+ "epoch": 0.61,
907
+ "eval_loss": 1.167747974395752,
908
+ "eval_runtime": 441.5379,
909
+ "eval_samples_per_second": 36.613,
910
+ "eval_steps_per_second": 1.146,
911
+ "step": 700
912
+ },
913
+ {
914
+ "epoch": 0.62,
915
+ "learning_rate": 7.664057699625215e-06,
916
+ "loss": 1.1644,
917
+ "step": 705
918
+ },
919
+ {
920
+ "epoch": 0.62,
921
+ "learning_rate": 7.515473829640987e-06,
922
+ "loss": 1.1369,
923
+ "step": 710
924
+ },
925
+ {
926
+ "epoch": 0.63,
927
+ "learning_rate": 7.367472304354011e-06,
928
+ "loss": 1.1445,
929
+ "step": 715
930
+ },
931
+ {
932
+ "epoch": 0.63,
933
+ "learning_rate": 7.2200878136397355e-06,
934
+ "loss": 1.1747,
935
+ "step": 720
936
+ },
937
+ {
938
+ "epoch": 0.64,
939
+ "learning_rate": 7.073354902747742e-06,
940
+ "loss": 1.1431,
941
+ "step": 725
942
+ },
943
+ {
944
+ "epoch": 0.64,
945
+ "learning_rate": 6.927307964204695e-06,
946
+ "loss": 1.1846,
947
+ "step": 730
948
+ },
949
+ {
950
+ "epoch": 0.64,
951
+ "learning_rate": 6.781981229753145e-06,
952
+ "loss": 1.1693,
953
+ "step": 735
954
+ },
955
+ {
956
+ "epoch": 0.65,
957
+ "learning_rate": 6.637408762327972e-06,
958
+ "loss": 1.1575,
959
+ "step": 740
960
+ },
961
+ {
962
+ "epoch": 0.65,
963
+ "learning_rate": 6.4936244480724575e-06,
964
+ "loss": 1.1708,
965
+ "step": 745
966
+ },
967
+ {
968
+ "epoch": 0.66,
969
+ "learning_rate": 6.350661988395723e-06,
970
+ "loss": 1.156,
971
+ "step": 750
972
+ },
973
+ {
974
+ "epoch": 0.66,
975
+ "learning_rate": 6.208554892073528e-06,
976
+ "loss": 1.1704,
977
+ "step": 755
978
+ },
979
+ {
980
+ "epoch": 0.67,
981
+ "learning_rate": 6.067336467394169e-06,
982
+ "loss": 1.1596,
983
+ "step": 760
984
+ },
985
+ {
986
+ "epoch": 0.67,
987
+ "learning_rate": 5.927039814351426e-06,
988
+ "loss": 1.1438,
989
+ "step": 765
990
+ },
991
+ {
992
+ "epoch": 0.67,
993
+ "learning_rate": 5.787697816886273e-06,
994
+ "loss": 1.1443,
995
+ "step": 770
996
+ },
997
+ {
998
+ "epoch": 0.68,
999
+ "learning_rate": 5.649343135179271e-06,
1000
+ "loss": 1.1509,
1001
+ "step": 775
1002
+ },
1003
+ {
1004
+ "epoch": 0.68,
1005
+ "learning_rate": 5.512008197995379e-06,
1006
+ "loss": 1.1432,
1007
+ "step": 780
1008
+ },
1009
+ {
1010
+ "epoch": 0.69,
1011
+ "learning_rate": 5.375725195083046e-06,
1012
+ "loss": 1.1613,
1013
+ "step": 785
1014
+ },
1015
+ {
1016
+ "epoch": 0.69,
1017
+ "learning_rate": 5.240526069629265e-06,
1018
+ "loss": 1.1385,
1019
+ "step": 790
1020
+ },
1021
+ {
1022
+ "epoch": 0.7,
1023
+ "learning_rate": 5.106442510772489e-06,
1024
+ "loss": 1.1672,
1025
+ "step": 795
1026
+ },
1027
+ {
1028
+ "epoch": 0.7,
1029
+ "learning_rate": 4.97350594617502e-06,
1030
+ "loss": 1.1531,
1031
+ "step": 800
1032
+ },
1033
+ {
1034
+ "epoch": 0.7,
1035
+ "eval_loss": 1.1635608673095703,
1036
+ "eval_runtime": 440.8581,
1037
+ "eval_samples_per_second": 36.669,
1038
+ "eval_steps_per_second": 1.148,
1039
+ "step": 800
1040
+ },
1041
+ {
1042
+ "epoch": 0.71,
1043
+ "learning_rate": 4.8417475346567635e-06,
1044
+ "loss": 1.138,
1045
+ "step": 805
1046
+ },
1047
+ {
1048
+ "epoch": 0.71,
1049
+ "learning_rate": 4.711198158891909e-06,
1050
+ "loss": 1.1437,
1051
+ "step": 810
1052
+ },
1053
+ {
1054
+ "epoch": 0.71,
1055
+ "learning_rate": 4.581888418170429e-06,
1056
+ "loss": 1.141,
1057
+ "step": 815
1058
+ },
1059
+ {
1060
+ "epoch": 0.72,
1061
+ "learning_rate": 4.453848621225913e-06,
1062
+ "loss": 1.1341,
1063
+ "step": 820
1064
+ },
1065
+ {
1066
+ "epoch": 0.72,
1067
+ "learning_rate": 4.327108779131573e-06,
1068
+ "loss": 1.1565,
1069
+ "step": 825
1070
+ },
1071
+ {
1072
+ "epoch": 0.73,
1073
+ "learning_rate": 4.201698598265973e-06,
1074
+ "loss": 1.1516,
1075
+ "step": 830
1076
+ },
1077
+ {
1078
+ "epoch": 0.73,
1079
+ "learning_rate": 4.077647473350201e-06,
1080
+ "loss": 1.1764,
1081
+ "step": 835
1082
+ },
1083
+ {
1084
+ "epoch": 0.74,
1085
+ "learning_rate": 3.954984480558071e-06,
1086
+ "loss": 1.1538,
1087
+ "step": 840
1088
+ },
1089
+ {
1090
+ "epoch": 0.74,
1091
+ "learning_rate": 3.83373837070101e-06,
1092
+ "loss": 1.1691,
1093
+ "step": 845
1094
+ },
1095
+ {
1096
+ "epoch": 0.74,
1097
+ "learning_rate": 3.7139375624891795e-06,
1098
+ "loss": 1.1479,
1099
+ "step": 850
1100
+ },
1101
+ {
1102
+ "epoch": 0.75,
1103
+ "learning_rate": 3.595610135870472e-06,
1104
+ "loss": 1.1609,
1105
+ "step": 855
1106
+ },
1107
+ {
1108
+ "epoch": 0.75,
1109
+ "learning_rate": 3.478783825448869e-06,
1110
+ "loss": 1.1556,
1111
+ "step": 860
1112
+ },
1113
+ {
1114
+ "epoch": 0.76,
1115
+ "learning_rate": 3.3634860139837877e-06,
1116
+ "loss": 1.1711,
1117
+ "step": 865
1118
+ },
1119
+ {
1120
+ "epoch": 0.76,
1121
+ "learning_rate": 3.249743725971849e-06,
1122
+ "loss": 1.1623,
1123
+ "step": 870
1124
+ },
1125
+ {
1126
+ "epoch": 0.77,
1127
+ "learning_rate": 3.1375836213126653e-06,
1128
+ "loss": 1.144,
1129
+ "step": 875
1130
+ },
1131
+ {
1132
+ "epoch": 0.77,
1133
+ "learning_rate": 3.0270319890600465e-06,
1134
+ "loss": 1.1572,
1135
+ "step": 880
1136
+ },
1137
+ {
1138
+ "epoch": 0.78,
1139
+ "learning_rate": 2.918114741260156e-06,
1140
+ "loss": 1.1427,
1141
+ "step": 885
1142
+ },
1143
+ {
1144
+ "epoch": 0.78,
1145
+ "learning_rate": 2.8108574068780093e-06,
1146
+ "loss": 1.1427,
1147
+ "step": 890
1148
+ },
1149
+ {
1150
+ "epoch": 0.78,
1151
+ "learning_rate": 2.7052851258137936e-06,
1152
+ "loss": 1.1488,
1153
+ "step": 895
1154
+ },
1155
+ {
1156
+ "epoch": 0.79,
1157
+ "learning_rate": 2.601422643010335e-06,
1158
+ "loss": 1.1525,
1159
+ "step": 900
1160
+ },
1161
+ {
1162
+ "epoch": 0.79,
1163
+ "eval_loss": 1.1609553098678589,
1164
+ "eval_runtime": 441.3737,
1165
+ "eval_samples_per_second": 36.627,
1166
+ "eval_steps_per_second": 1.146,
1167
+ "step": 900
1168
+ },
1169
+ {
1170
+ "epoch": 0.79,
1171
+ "learning_rate": 2.4992943026531935e-06,
1172
+ "loss": 1.1582,
1173
+ "step": 905
1174
+ },
1175
+ {
1176
+ "epoch": 0.8,
1177
+ "learning_rate": 2.3989240424646355e-06,
1178
+ "loss": 1.1551,
1179
+ "step": 910
1180
+ },
1181
+ {
1182
+ "epoch": 0.8,
1183
+ "learning_rate": 2.300335388092929e-06,
1184
+ "loss": 1.1683,
1185
+ "step": 915
1186
+ },
1187
+ {
1188
+ "epoch": 0.81,
1189
+ "learning_rate": 2.2035514475981756e-06,
1190
+ "loss": 1.1447,
1191
+ "step": 920
1192
+ },
1193
+ {
1194
+ "epoch": 0.81,
1195
+ "learning_rate": 2.1085949060360654e-06,
1196
+ "loss": 1.1543,
1197
+ "step": 925
1198
+ },
1199
+ {
1200
+ "epoch": 0.81,
1201
+ "learning_rate": 2.015488020140737e-06,
1202
+ "loss": 1.1448,
1203
+ "step": 930
1204
+ },
1205
+ {
1206
+ "epoch": 0.82,
1207
+ "learning_rate": 1.924252613108073e-06,
1208
+ "loss": 1.1677,
1209
+ "step": 935
1210
+ },
1211
+ {
1212
+ "epoch": 0.82,
1213
+ "learning_rate": 1.8349100694805711e-06,
1214
+ "loss": 1.1586,
1215
+ "step": 940
1216
+ },
1217
+ {
1218
+ "epoch": 0.83,
1219
+ "learning_rate": 1.7474813301350668e-06,
1220
+ "loss": 1.1592,
1221
+ "step": 945
1222
+ },
1223
+ {
1224
+ "epoch": 0.83,
1225
+ "learning_rate": 1.661986887374415e-06,
1226
+ "loss": 1.1702,
1227
+ "step": 950
1228
+ },
1229
+ {
1230
+ "epoch": 0.84,
1231
+ "learning_rate": 1.578446780124344e-06,
1232
+ "loss": 1.1667,
1233
+ "step": 955
1234
+ },
1235
+ {
1236
+ "epoch": 0.84,
1237
+ "learning_rate": 1.49688058923654e-06,
1238
+ "loss": 1.1489,
1239
+ "step": 960
1240
+ },
1241
+ {
1242
+ "epoch": 0.85,
1243
+ "learning_rate": 1.4173074328991376e-06,
1244
+ "loss": 1.1448,
1245
+ "step": 965
1246
+ },
1247
+ {
1248
+ "epoch": 0.85,
1249
+ "learning_rate": 1.339745962155613e-06,
1250
+ "loss": 1.1531,
1251
+ "step": 970
1252
+ },
1253
+ {
1254
+ "epoch": 0.85,
1255
+ "learning_rate": 1.2642143565332154e-06,
1256
+ "loss": 1.1668,
1257
+ "step": 975
1258
+ },
1259
+ {
1260
+ "epoch": 0.86,
1261
+ "learning_rate": 1.1907303197818665e-06,
1262
+ "loss": 1.1618,
1263
+ "step": 980
1264
+ },
1265
+ {
1266
+ "epoch": 0.86,
1267
+ "learning_rate": 1.1193110757246251e-06,
1268
+ "loss": 1.1456,
1269
+ "step": 985
1270
+ },
1271
+ {
1272
+ "epoch": 0.87,
1273
+ "learning_rate": 1.0499733642206034e-06,
1274
+ "loss": 1.1493,
1275
+ "step": 990
1276
+ },
1277
+ {
1278
+ "epoch": 0.87,
1279
+ "learning_rate": 9.827334372413444e-07,
1280
+ "loss": 1.1657,
1281
+ "step": 995
1282
+ },
1283
+ {
1284
+ "epoch": 0.88,
1285
+ "learning_rate": 9.176070550615379e-07,
1286
+ "loss": 1.1396,
1287
+ "step": 1000
1288
+ },
1289
+ {
1290
+ "epoch": 0.88,
1291
+ "eval_loss": 1.1596382856369019,
1292
+ "eval_runtime": 440.1747,
1293
+ "eval_samples_per_second": 36.726,
1294
+ "eval_steps_per_second": 1.15,
1295
+ "step": 1000
1296
+ },
1297
+ {
1298
+ "epoch": 0.88,
1299
+ "learning_rate": 8.546094825649909e-07,
1300
+ "loss": 1.1409,
1301
+ "step": 1005
1302
+ },
1303
+ {
1304
+ "epoch": 0.88,
1305
+ "learning_rate": 7.937554856667196e-07,
1306
+ "loss": 1.1476,
1307
+ "step": 1010
1308
+ },
1309
+ {
1310
+ "epoch": 0.89,
1311
+ "learning_rate": 7.350593278519824e-07,
1312
+ "loss": 1.1424,
1313
+ "step": 1015
1314
+ },
1315
+ {
1316
+ "epoch": 0.89,
1317
+ "learning_rate": 6.785347668330777e-07,
1318
+ "loss": 1.1394,
1319
+ "step": 1020
1320
+ },
1321
+ {
1322
+ "epoch": 0.9,
1323
+ "learning_rate": 6.241950513246931e-07,
1324
+ "loss": 1.159,
1325
+ "step": 1025
1326
+ },
1327
+ {
1328
+ "epoch": 0.9,
1329
+ "learning_rate": 5.720529179385659e-07,
1330
+ "loss": 1.139,
1331
+ "step": 1030
1332
+ },
1333
+ {
1334
+ "epoch": 0.91,
1335
+ "learning_rate": 5.221205881981594e-07,
1336
+ "loss": 1.15,
1337
+ "step": 1035
1338
+ },
1339
+ {
1340
+ "epoch": 0.91,
1341
+ "learning_rate": 4.7440976567407096e-07,
1342
+ "loss": 1.1592,
1343
+ "step": 1040
1344
+ },
1345
+ {
1346
+ "epoch": 0.92,
1347
+ "learning_rate": 4.2893163324085886e-07,
1348
+ "loss": 1.1574,
1349
+ "step": 1045
1350
+ },
1351
+ {
1352
+ "epoch": 0.92,
1353
+ "learning_rate": 3.856968504558989e-07,
1354
+ "loss": 1.1491,
1355
+ "step": 1050
1356
+ },
1357
+ {
1358
+ "epoch": 0.92,
1359
+ "learning_rate": 3.4471555106090573e-07,
1360
+ "loss": 1.1452,
1361
+ "step": 1055
1362
+ },
1363
+ {
1364
+ "epoch": 0.93,
1365
+ "learning_rate": 3.059973406066963e-07,
1366
+ "loss": 1.1886,
1367
+ "step": 1060
1368
+ },
1369
+ {
1370
+ "epoch": 0.93,
1371
+ "learning_rate": 2.6955129420176193e-07,
1372
+ "loss": 1.1573,
1373
+ "step": 1065
1374
+ },
1375
+ {
1376
+ "epoch": 0.94,
1377
+ "learning_rate": 2.3538595438516442e-07,
1378
+ "loss": 1.1489,
1379
+ "step": 1070
1380
+ },
1381
+ {
1382
+ "epoch": 0.94,
1383
+ "learning_rate": 2.035093291242607e-07,
1384
+ "loss": 1.1433,
1385
+ "step": 1075
1386
+ },
1387
+ {
1388
+ "epoch": 0.95,
1389
+ "learning_rate": 1.7392888993773005e-07,
1390
+ "loss": 1.1521,
1391
+ "step": 1080
1392
+ },
1393
+ {
1394
+ "epoch": 0.95,
1395
+ "learning_rate": 1.466515701443294e-07,
1396
+ "loss": 1.1386,
1397
+ "step": 1085
1398
+ },
1399
+ {
1400
+ "epoch": 0.95,
1401
+ "learning_rate": 1.2168376323780652e-07,
1402
+ "loss": 1.1377,
1403
+ "step": 1090
1404
+ },
1405
+ {
1406
+ "epoch": 0.96,
1407
+ "learning_rate": 9.90313213883376e-08,
1408
+ "loss": 1.1554,
1409
+ "step": 1095
1410
+ },
1411
+ {
1412
+ "epoch": 0.96,
1413
+ "learning_rate": 7.86995540708424e-08,
1414
+ "loss": 1.1681,
1415
+ "step": 1100
1416
+ },
1417
+ {
1418
+ "epoch": 0.96,
1419
+ "eval_loss": 1.159261703491211,
1420
+ "eval_runtime": 441.2771,
1421
+ "eval_samples_per_second": 36.635,
1422
+ "eval_steps_per_second": 1.147,
1423
+ "step": 1100
1424
+ },
1425
+ {
1426
+ "epoch": 0.97,
1427
+ "learning_rate": 6.069322682050516e-08,
1428
+ "loss": 1.1621,
1429
+ "step": 1105
1430
+ },
1431
+ {
1432
+ "epoch": 0.97,
1433
+ "learning_rate": 4.501656011579037e-08,
1434
+ "loss": 1.135,
1435
+ "step": 1110
1436
+ },
1437
+ {
1438
+ "epoch": 0.98,
1439
+ "learning_rate": 3.167322838920406e-08,
1440
+ "loss": 1.1301,
1441
+ "step": 1115
1442
+ },
1443
+ {
1444
+ "epoch": 0.98,
1445
+ "learning_rate": 2.066635916605386e-08,
1446
+ "loss": 1.1462,
1447
+ "step": 1120
1448
+ },
1449
+ {
1450
+ "epoch": 0.99,
1451
+ "learning_rate": 1.1998532331389812e-08,
1452
+ "loss": 1.1398,
1453
+ "step": 1125
1454
+ },
1455
+ {
1456
+ "epoch": 0.99,
1457
+ "learning_rate": 5.671779525311394e-09,
1458
+ "loss": 1.1599,
1459
+ "step": 1130
1460
+ },
1461
+ {
1462
+ "epoch": 0.99,
1463
+ "learning_rate": 1.6875836667729073e-09,
1464
+ "loss": 1.1463,
1465
+ "step": 1135
1466
+ },
1467
+ {
1468
+ "epoch": 1.0,
1469
+ "learning_rate": 4.687860599927874e-11,
1470
+ "loss": 1.1576,
1471
+ "step": 1140
1472
+ },
1473
+ {
1474
+ "epoch": 1.0,
1475
+ "step": 1141,
1476
+ "total_flos": 1.1420601686146679e+19,
1477
+ "train_loss": 1.2278979039839963,
1478
+ "train_runtime": 17901.7657,
1479
+ "train_samples_per_second": 8.159,
1480
+ "train_steps_per_second": 0.064
1481
+ }
1482
+ ],
1483
+ "logging_steps": 5,
1484
+ "max_steps": 1141,
1485
+ "num_input_tokens_seen": 0,
1486
+ "num_train_epochs": 1,
1487
+ "save_steps": 100,
1488
+ "total_flos": 1.1420601686146679e+19,
1489
+ "train_batch_size": 16,
1490
+ "trial_name": null,
1491
+ "trial_params": null
1492
+ }