Hugging Face's logo

LoneStriker
/
notux-8x7b-v1-2.4bpw-h6-exl2

Text Generation
Transformers
Safetensors
mixtral
dpo
rlaif
preference
ultrafeedback
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
LoneStriker commited on
Commit
80d4ffa
•
1 Parent(s): 3a6cc5c

Upload folder using huggingface_hub

Browse files
Files changed (14) hide show
  1. README.md +96 -0
  2. all_results.json +21 -0
  3. config.json +30 -0
  4. eval_results.json +16 -0
  5. generation_config.json +6 -0
  6. huggingface-metadata.txt +23 -0
  7. model.safetensors.index.json +1002 -0
  8. output-00001-of-00002.safetensors +3 -0
  9. output-00002-of-00002.safetensors +3 -0
  10. special_tokens_map.json +24 -0
  11. tokenizer.json +0 -0
  12. tokenizer_config.json +43 -0
  13. train_results.json +8 -0
  14. trainer_state.json +1396 -0
README.md ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - argilla/ultrafeedback-binarized-preferences-cleaned
4
+ language:
5
+ - en
6
+ - de
7
+ - es
8
+ - fr
9
+ - it
10
+ base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
11
+ library_name: transformers
12
+ pipeline_tag: text-generation
13
+ tags:
14
+ - dpo
15
+ - rlaif
16
+ - preference
17
+ - ultrafeedback
18
+ license: apache-2.0
19
+ model-index:
20
+ - name: notux-8x7b-v1
21
+ results: []
22
+ ---
23
+
24
+ <div align="center">
25
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/60f0608166e5701b80ed3f02/dj-spsk9eXMMXVGxK6jRz.png" alt="A banner representing Notus, the wind god of the south, in a mythical and artistic style. The banner features a strong, swirling breeze, embodying the warm, wet character of the southern wind. Gracefully flowing across the scene are several paper planes, caught in the gentle yet powerful gusts of Notus. The background is a blend of warm colors, symbolizing the heat of the south, with hints of blue and green to represent the moisture carried by this wind. The overall atmosphere is one of dynamic movement and warmth."/>
26
+ </div>
27
+
28
+
29
+ # Model Card for Notux 8x7B-v1
30
+
31
+ This model is a preference-tuned version of [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) on the [argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned) dataset using DPO (Direct Preference Optimization).
32
+
33
+ As of Dec 26th 2023, it outperforms `Mixtral-8x7B-Instruct-v0.1` and is the top ranked MoE (Mixture of Experts) model on the [Hugging Face Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
34
+
35
+ This is part of the Notus family of models and experiments, where the Argilla team investigates data-first and preference tuning methods like dDPO (distilled DPO). This model is the result of our first experiment at tuning a MoE model that has already been fine-tuned with DPO (i.e., Mixtral-8x7B-Instruct-v0.1).
36
+
37
+ ## Model Details
38
+
39
+ ### Model Description
40
+
41
+ - **Developed by:** Argilla (based on MistralAI previous efforts)
42
+ - **Shared by:** Argilla
43
+ - **Model type:** Pretrained generative Sparse Mixture of Experts
44
+ - **Language(s) (NLP):** English, Spanish, Italian, German, and French
45
+ - **License:** MIT
46
+ - **Finetuned from model:** [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
47
+
48
+ ### Model Sources
49
+
50
+ - **Repository:** https://github.com/argilla-io/notus
51
+ - **Paper:** N/A
52
+
53
+ ## Training Details
54
+
55
+ ### Training Hardware
56
+
57
+ We used a VM with 8 x H100 80GB hosted in runpod.io for 1 epoch (~10hr).
58
+
59
+ ### Training Data
60
+
61
+ We used a new iteration of the Argilla UltraFeedback preferences dataset named [argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned).
62
+
63
+ ## Training procedure
64
+
65
+ ### Training hyperparameters
66
+
67
+ The following hyperparameters were used during training:
68
+ - learning_rate: 5e-07
69
+ - train_batch_size: 8
70
+ - eval_batch_size: 4
71
+ - seed: 42
72
+ - distributed_type: multi-GPU
73
+ - num_devices: 8
74
+ - total_train_batch_size: 64
75
+ - total_eval_batch_size: 32
76
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
77
+ - lr_scheduler_type: linear
78
+ - lr_scheduler_warmup_ratio: 0.1
79
+ - num_epochs: 1
80
+
81
+ ### Training results
82
+
83
+ | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
84
+ |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
85
+ | 0.4384 | 0.22 | 200 | 0.4556 | -0.3275 | -1.9448 | 0.7937 | 1.6174 | -405.7994 | -397.8617 | -1.3157 | -1.4511 |
86
+ | 0.4064 | 0.43 | 400 | 0.4286 | -0.2163 | -2.2090 | 0.8254 | 1.9927 | -408.4409 | -396.7496 | -0.7660 | -0.6539 |
87
+ | 0.3952 | 0.65 | 600 | 0.4275 | -0.1311 | -2.1603 | 0.8016 | 2.0291 | -407.9537 | -395.8982 | -0.6783 | -0.7206 |
88
+ | 0.3909 | 0.87 | 800 | 0.4167 | -0.2273 | -2.3146 | 0.8135 | 2.0872 | -409.4968 | -396.8602 | -0.8458 | -0.7738 |
89
+
90
+
91
+ ### Framework versions
92
+
93
+ - Transformers 4.36.0
94
+ - Pytorch 2.1.0+cu118
95
+ - Datasets 2.14.6
96
+ - Tokenizers 0.15.0
all_results.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_logits/chosen": -1.2132024765014648,
4
+ "eval_logits/rejected": -1.2924575805664062,
5
+ "eval_logps/chosen": -396.52020263671875,
6
+ "eval_logps/rejected": -409.31964111328125,
7
+ "eval_loss": 0.4216844439506531,
8
+ "eval_rewards/accuracies": 0.8134920597076416,
9
+ "eval_rewards/chosen": -0.1933162659406662,
10
+ "eval_rewards/margins": 2.1035311222076416,
11
+ "eval_rewards/rejected": -2.296847343444824,
12
+ "eval_runtime": 398.9749,
13
+ "eval_samples": 2000,
14
+ "eval_samples_per_second": 5.013,
15
+ "eval_steps_per_second": 0.158,
16
+ "train_loss": 0.4461688995361328,
17
+ "train_runtime": 44067.2139,
18
+ "train_samples": 58917,
19
+ "train_samples_per_second": 1.337,
20
+ "train_steps_per_second": 0.021
21
+ }
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "mistralai/Mixtral-8x7B-Instruct-v0.1",
3
+ "architectures": [
4
+ "MixtralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mixtral",
15
+ "num_attention_heads": 32,
16
+ "num_experts_per_tok": 2,
17
+ "num_hidden_layers": 32,
18
+ "num_key_value_heads": 8,
19
+ "num_local_experts": 8,
20
+ "output_router_logits": false,
21
+ "rms_norm_eps": 1e-05,
22
+ "rope_theta": 1000000.0,
23
+ "router_aux_loss_coef": 0.02,
24
+ "sliding_window": 4096,
25
+ "tie_word_embeddings": false,
26
+ "torch_dtype": "bfloat16",
27
+ "transformers_version": "4.36.0",
28
+ "use_cache": false,
29
+ "vocab_size": 32000
30
+ }
eval_results.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_logits/chosen": -1.2132024765014648,
4
+ "eval_logits/rejected": -1.2924575805664062,
5
+ "eval_logps/chosen": -396.52020263671875,
6
+ "eval_logps/rejected": -409.31964111328125,
7
+ "eval_loss": 0.4216844439506531,
8
+ "eval_rewards/accuracies": 0.8134920597076416,
9
+ "eval_rewards/chosen": -0.1933162659406662,
10
+ "eval_rewards/margins": 2.1035311222076416,
11
+ "eval_rewards/rejected": -2.296847343444824,
12
+ "eval_runtime": 398.9749,
13
+ "eval_samples": 2000,
14
+ "eval_samples_per_second": 5.013,
15
+ "eval_steps_per_second": 0.158
16
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.36.0"
6
+ }
huggingface-metadata.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ url: https://huggingface.co/argilla/notux-8x7b-v1
2
+ branch: main
3
+ download date: 2023-12-28 09:14:34
4
+ sha256sum:
5
+ 11f2ec2e11347fa0c374b236801fe3d432e30c900905c61483ae9f189dd5808d model-00001-of-00019.safetensors
6
+ 49ea4c94bf1abd460b2c09392b946e58e129e1e3a53ed702ce06d13b764bac39 model-00002-of-00019.safetensors
7
+ 5ada743d60cf3c5df04a9dace1bb9fb5fd16c1352dddcfa5e1e8a22b2e8a585f model-00003-of-00019.safetensors
8
+ d7496f8eefdda8a98946e32c07b16650ea23186363ccfa6afced76e6c1e15ae5 model-00004-of-00019.safetensors
9
+ 648bdaa1a215f57aea8cc29e66dc22b9a3d23ddab6ea86bba491622c7bfa9425 model-00005-of-00019.safetensors
10
+ d9668bb97c9c15c8496bc738549d2ff65c9f408869f8b673a8dc44f7ccf07be0 model-00006-of-00019.safetensors
11
+ 289504c1cd644785db1253537488ce04828517a4b3fa1dd9c3f3696ac5489d74 model-00007-of-00019.safetensors
12
+ 723a2545cb4f3b03d282ac0a914b4463f861c2f3a899c3e590227e185a866876 model-00008-of-00019.safetensors
13
+ bd4b0f8cb00843bae0d0c9c4135039abcdf82d6ce5dc5962176b0fa5e1a2a1b0 model-00009-of-00019.safetensors
14
+ 9ecfb4d51efb4e545d8c5f941e1cff6ddedcc21753a4616012f04e887ff4a97a model-00010-of-00019.safetensors
15
+ f2d22020018d58f1daee771c300911bd7d39ab7a1e8341f3386f4dc28b0191d3 model-00011-of-00019.safetensors
16
+ 8a24f06bdb175d79e93e29040b37da98e786c7195675bd2f5da28e07484bdb6a model-00012-of-00019.safetensors
17
+ 9da22cbd79d759e7d5c0bd61faca98443e18ad77154c777901d1b0ebc68503ae model-00013-of-00019.safetensors
18
+ dfdfac6da5c5f503367f73cc6aed1b86e0968ae0e5cad46cdee1a79bb1c4481e model-00014-of-00019.safetensors
19
+ 5f583fc17b1c5230b0cfbf5571aa275d044828aa641cbbaf3b3eec800975e8c4 model-00015-of-00019.safetensors
20
+ 3da333874351b8edd12e6a42cdf616c3cae0841ec86f12fa29f5e6c70d1b39b0 model-00016-of-00019.safetensors
21
+ aba2d6a68b8ff90041579d8ce6713e7487c9f4c31f4e181fb08b35299de7b45b model-00017-of-00019.safetensors
22
+ 37f26b990e1ee7a3db9a41c78de21e9edb1e1ee7d46f853d6a1931460d72a38d model-00018-of-00019.safetensors
23
+ cd4c7992df71e2366b507506231eaa6469fed4517fb912bd846f5b0c9f678195 model-00019-of-00019.safetensors
model.safetensors.index.json ADDED
@@ -0,0 +1,1002 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 93405585408
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00019-of-00019.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00019.safetensors",
8
+ "model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00019.safetensors",
9
+ "model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00019.safetensors",
10
+ "model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00019.safetensors",
11
+ "model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00019.safetensors",
12
+ "model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00019.safetensors",
13
+ "model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00019.safetensors",
14
+ "model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00019.safetensors",
15
+ "model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00019.safetensors",
16
+ "model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00019.safetensors",
17
+ "model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00019.safetensors",
18
+ "model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00019.safetensors",
19
+ "model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00019.safetensors",
20
+ "model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00019.safetensors",
21
+ "model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00019.safetensors",
22
+ "model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00019.safetensors",
23
+ "model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00019.safetensors",
24
+ "model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00019.safetensors",
25
+ "model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00019.safetensors",
26
+ "model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00019.safetensors",
27
+ "model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00019.safetensors",
28
+ "model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00019.safetensors",
29
+ "model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00019.safetensors",
30
+ "model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00019.safetensors",
31
+ "model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00019.safetensors",
32
+ "model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00019.safetensors",
33
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00019.safetensors",
34
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00019.safetensors",
35
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00019.safetensors",
36
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00019.safetensors",
37
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00019.safetensors",
38
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00019.safetensors",
39
+ "model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00019.safetensors",
40
+ "model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00019.safetensors",
41
+ "model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00019.safetensors",
42
+ "model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00019.safetensors",
43
+ "model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00019.safetensors",
44
+ "model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00019.safetensors",
45
+ "model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00019.safetensors",
46
+ "model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00019.safetensors",
47
+ "model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00019.safetensors",
48
+ "model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00019.safetensors",
49
+ "model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00019.safetensors",
50
+ "model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00019.safetensors",
51
+ "model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00019.safetensors",
52
+ "model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00019.safetensors",
53
+ "model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00019.safetensors",
54
+ "model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00019.safetensors",
55
+ "model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00019.safetensors",
56
+ "model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00019.safetensors",
57
+ "model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00019.safetensors",
58
+ "model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00019.safetensors",
59
+ "model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00019.safetensors",
60
+ "model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00019.safetensors",
61
+ "model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00019.safetensors",
62
+ "model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00019.safetensors",
63
+ "model.layers.1.block_sparse_moe.gate.weight": "model-00001-of-00019.safetensors",
64
+ "model.layers.1.input_layernorm.weight": "model-00002-of-00019.safetensors",
65
+ "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00019.safetensors",
66
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00019.safetensors",
67
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00019.safetensors",
68
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00019.safetensors",
69
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00019.safetensors",
70
+ "model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00019.safetensors",
71
+ "model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00019.safetensors",
72
+ "model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00019.safetensors",
73
+ "model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00019.safetensors",
74
+ "model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00007-of-00019.safetensors",
75
+ "model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00019.safetensors",
76
+ "model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00019.safetensors",
77
+ "model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00019.safetensors",
78
+ "model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00019.safetensors",
79
+ "model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00019.safetensors",
80
+ "model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00019.safetensors",
81
+ "model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00019.safetensors",
82
+ "model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00019.safetensors",
83
+ "model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00019.safetensors",
84
+ "model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00019.safetensors",
85
+ "model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00019.safetensors",
86
+ "model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00019.safetensors",
87
+ "model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00019.safetensors",
88
+ "model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00019.safetensors",
89
+ "model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00007-of-00019.safetensors",
90
+ "model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00007-of-00019.safetensors",
91
+ "model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00007-of-00019.safetensors",
92
+ "model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00007-of-00019.safetensors",
93
+ "model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00007-of-00019.safetensors",
94
+ "model.layers.10.block_sparse_moe.gate.weight": "model-00006-of-00019.safetensors",
95
+ "model.layers.10.input_layernorm.weight": "model-00007-of-00019.safetensors",
96
+ "model.layers.10.post_attention_layernorm.weight": "model-00007-of-00019.safetensors",
97
+ "model.layers.10.self_attn.k_proj.weight": "model-00006-of-00019.safetensors",
98
+ "model.layers.10.self_attn.o_proj.weight": "model-00006-of-00019.safetensors",
99
+ "model.layers.10.self_attn.q_proj.weight": "model-00006-of-00019.safetensors",
100
+ "model.layers.10.self_attn.v_proj.weight": "model-00006-of-00019.safetensors",
101
+ "model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00007-of-00019.safetensors",
102
+ "model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00007-of-00019.safetensors",
103
+ "model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00007-of-00019.safetensors",
104
+ "model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00019.safetensors",
105
+ "model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00007-of-00019.safetensors",
106
+ "model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00019.safetensors",
107
+ "model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00019.safetensors",
108
+ "model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00019.safetensors",
109
+ "model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00019.safetensors",
110
+ "model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00019.safetensors",
111
+ "model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00019.safetensors",
112
+ "model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00019.safetensors",
113
+ "model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00019.safetensors",
114
+ "model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00019.safetensors",
115
+ "model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00019.safetensors",
116
+ "model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00019.safetensors",
117
+ "model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00019.safetensors",
118
+ "model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00019.safetensors",
119
+ "model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00019.safetensors",
120
+ "model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00007-of-00019.safetensors",
121
+ "model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00008-of-00019.safetensors",
122
+ "model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00008-of-00019.safetensors",
123
+ "model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00008-of-00019.safetensors",
124
+ "model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00008-of-00019.safetensors",
125
+ "model.layers.11.block_sparse_moe.gate.weight": "model-00007-of-00019.safetensors",
126
+ "model.layers.11.input_layernorm.weight": "model-00008-of-00019.safetensors",
127
+ "model.layers.11.post_attention_layernorm.weight": "model-00008-of-00019.safetensors",
128
+ "model.layers.11.self_attn.k_proj.weight": "model-00007-of-00019.safetensors",
129
+ "model.layers.11.self_attn.o_proj.weight": "model-00007-of-00019.safetensors",
130
+ "model.layers.11.self_attn.q_proj.weight": "model-00007-of-00019.safetensors",
131
+ "model.layers.11.self_attn.v_proj.weight": "model-00007-of-00019.safetensors",
132
+ "model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00008-of-00019.safetensors",
133
+ "model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00019.safetensors",
134
+ "model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00008-of-00019.safetensors",
135
+ "model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00008-of-00019.safetensors",
136
+ "model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00019.safetensors",
137
+ "model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00008-of-00019.safetensors",
138
+ "model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00008-of-00019.safetensors",
139
+ "model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00019.safetensors",
140
+ "model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00008-of-00019.safetensors",
141
+ "model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00008-of-00019.safetensors",
142
+ "model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00019.safetensors",
143
+ "model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00008-of-00019.safetensors",
144
+ "model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00008-of-00019.safetensors",
145
+ "model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00008-of-00019.safetensors",
146
+ "model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00008-of-00019.safetensors",
147
+ "model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00008-of-00019.safetensors",
148
+ "model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00008-of-00019.safetensors",
149
+ "model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00008-of-00019.safetensors",
150
+ "model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00008-of-00019.safetensors",
151
+ "model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00008-of-00019.safetensors",
152
+ "model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00008-of-00019.safetensors",
153
+ "model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00008-of-00019.safetensors",
154
+ "model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00008-of-00019.safetensors",
155
+ "model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00008-of-00019.safetensors",
156
+ "model.layers.12.block_sparse_moe.gate.weight": "model-00008-of-00019.safetensors",
157
+ "model.layers.12.input_layernorm.weight": "model-00008-of-00019.safetensors",
158
+ "model.layers.12.post_attention_layernorm.weight": "model-00008-of-00019.safetensors",
159
+ "model.layers.12.self_attn.k_proj.weight": "model-00008-of-00019.safetensors",
160
+ "model.layers.12.self_attn.o_proj.weight": "model-00008-of-00019.safetensors",
161
+ "model.layers.12.self_attn.q_proj.weight": "model-00008-of-00019.safetensors",
162
+ "model.layers.12.self_attn.v_proj.weight": "model-00008-of-00019.safetensors",
163
+ "model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00008-of-00019.safetensors",
164
+ "model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00019.safetensors",
165
+ "model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00008-of-00019.safetensors",
166
+ "model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00008-of-00019.safetensors",
167
+ "model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00019.safetensors",
168
+ "model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00008-of-00019.safetensors",
169
+ "model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00008-of-00019.safetensors",
170
+ "model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00019.safetensors",
171
+ "model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00008-of-00019.safetensors",
172
+ "model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00008-of-00019.safetensors",
173
+ "model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00019.safetensors",
174
+ "model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00008-of-00019.safetensors",
175
+ "model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00008-of-00019.safetensors",
176
+ "model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00009-of-00019.safetensors",
177
+ "model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00009-of-00019.safetensors",
178
+ "model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00009-of-00019.safetensors",
179
+ "model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00009-of-00019.safetensors",
180
+ "model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00009-of-00019.safetensors",
181
+ "model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00009-of-00019.safetensors",
182
+ "model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00009-of-00019.safetensors",
183
+ "model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00009-of-00019.safetensors",
184
+ "model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00009-of-00019.safetensors",
185
+ "model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00009-of-00019.safetensors",
186
+ "model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00009-of-00019.safetensors",
187
+ "model.layers.13.block_sparse_moe.gate.weight": "model-00008-of-00019.safetensors",
188
+ "model.layers.13.input_layernorm.weight": "model-00009-of-00019.safetensors",
189
+ "model.layers.13.post_attention_layernorm.weight": "model-00009-of-00019.safetensors",
190
+ "model.layers.13.self_attn.k_proj.weight": "model-00008-of-00019.safetensors",
191
+ "model.layers.13.self_attn.o_proj.weight": "model-00008-of-00019.safetensors",
192
+ "model.layers.13.self_attn.q_proj.weight": "model-00008-of-00019.safetensors",
193
+ "model.layers.13.self_attn.v_proj.weight": "model-00008-of-00019.safetensors",
194
+ "model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00019.safetensors",
195
+ "model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00009-of-00019.safetensors",
196
+ "model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00019.safetensors",
197
+ "model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00019.safetensors",
198
+ "model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00009-of-00019.safetensors",
199
+ "model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00019.safetensors",
200
+ "model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00009-of-00019.safetensors",
201
+ "model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00009-of-00019.safetensors",
202
+ "model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00009-of-00019.safetensors",
203
+ "model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00009-of-00019.safetensors",
204
+ "model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00009-of-00019.safetensors",
205
+ "model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00009-of-00019.safetensors",
206
+ "model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00009-of-00019.safetensors",
207
+ "model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00009-of-00019.safetensors",
208
+ "model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00009-of-00019.safetensors",
209
+ "model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00009-of-00019.safetensors",
210
+ "model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00009-of-00019.safetensors",
211
+ "model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00009-of-00019.safetensors",
212
+ "model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00009-of-00019.safetensors",
213
+ "model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00009-of-00019.safetensors",
214
+ "model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00009-of-00019.safetensors",
215
+ "model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00009-of-00019.safetensors",
216
+ "model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00009-of-00019.safetensors",
217
+ "model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00009-of-00019.safetensors",
218
+ "model.layers.14.block_sparse_moe.gate.weight": "model-00009-of-00019.safetensors",
219
+ "model.layers.14.input_layernorm.weight": "model-00009-of-00019.safetensors",
220
+ "model.layers.14.post_attention_layernorm.weight": "model-00009-of-00019.safetensors",
221
+ "model.layers.14.self_attn.k_proj.weight": "model-00009-of-00019.safetensors",
222
+ "model.layers.14.self_attn.o_proj.weight": "model-00009-of-00019.safetensors",
223
+ "model.layers.14.self_attn.q_proj.weight": "model-00009-of-00019.safetensors",
224
+ "model.layers.14.self_attn.v_proj.weight": "model-00009-of-00019.safetensors",
225
+ "model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00019.safetensors",
226
+ "model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00009-of-00019.safetensors",
227
+ "model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00019.safetensors",
228
+ "model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00019.safetensors",
229
+ "model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00009-of-00019.safetensors",
230
+ "model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00019.safetensors",
231
+ "model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00010-of-00019.safetensors",
232
+ "model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00010-of-00019.safetensors",
233
+ "model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00010-of-00019.safetensors",
234
+ "model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00010-of-00019.safetensors",
235
+ "model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00010-of-00019.safetensors",
236
+ "model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00010-of-00019.safetensors",
237
+ "model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00010-of-00019.safetensors",
238
+ "model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00010-of-00019.safetensors",
239
+ "model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00010-of-00019.safetensors",
240
+ "model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00010-of-00019.safetensors",
241
+ "model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00010-of-00019.safetensors",
242
+ "model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00010-of-00019.safetensors",
243
+ "model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00010-of-00019.safetensors",
244
+ "model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00010-of-00019.safetensors",
245
+ "model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00010-of-00019.safetensors",
246
+ "model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00010-of-00019.safetensors",
247
+ "model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00010-of-00019.safetensors",
248
+ "model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00010-of-00019.safetensors",
249
+ "model.layers.15.block_sparse_moe.gate.weight": "model-00009-of-00019.safetensors",
250
+ "model.layers.15.input_layernorm.weight": "model-00010-of-00019.safetensors",
251
+ "model.layers.15.post_attention_layernorm.weight": "model-00010-of-00019.safetensors",
252
+ "model.layers.15.self_attn.k_proj.weight": "model-00009-of-00019.safetensors",
253
+ "model.layers.15.self_attn.o_proj.weight": "model-00009-of-00019.safetensors",
254
+ "model.layers.15.self_attn.q_proj.weight": "model-00009-of-00019.safetensors",
255
+ "model.layers.15.self_attn.v_proj.weight": "model-00009-of-00019.safetensors",
256
+ "model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00010-of-00019.safetensors",
257
+ "model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00010-of-00019.safetensors",
258
+ "model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00010-of-00019.safetensors",
259
+ "model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00010-of-00019.safetensors",
260
+ "model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00010-of-00019.safetensors",
261
+ "model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00010-of-00019.safetensors",
262
+ "model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00010-of-00019.safetensors",
263
+ "model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00010-of-00019.safetensors",
264
+ "model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00010-of-00019.safetensors",
265
+ "model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00010-of-00019.safetensors",
266
+ "model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00010-of-00019.safetensors",
267
+ "model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00010-of-00019.safetensors",
268
+ "model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00010-of-00019.safetensors",
269
+ "model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00010-of-00019.safetensors",
270
+ "model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00010-of-00019.safetensors",
271
+ "model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00010-of-00019.safetensors",
272
+ "model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00010-of-00019.safetensors",
273
+ "model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00010-of-00019.safetensors",
274
+ "model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00010-of-00019.safetensors",
275
+ "model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00010-of-00019.safetensors",
276
+ "model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00010-of-00019.safetensors",
277
+ "model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00010-of-00019.safetensors",
278
+ "model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00010-of-00019.safetensors",
279
+ "model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00011-of-00019.safetensors",
280
+ "model.layers.16.block_sparse_moe.gate.weight": "model-00010-of-00019.safetensors",
281
+ "model.layers.16.input_layernorm.weight": "model-00011-of-00019.safetensors",
282
+ "model.layers.16.post_attention_layernorm.weight": "model-00011-of-00019.safetensors",
283
+ "model.layers.16.self_attn.k_proj.weight": "model-00010-of-00019.safetensors",
284
+ "model.layers.16.self_attn.o_proj.weight": "model-00010-of-00019.safetensors",
285
+ "model.layers.16.self_attn.q_proj.weight": "model-00010-of-00019.safetensors",
286
+ "model.layers.16.self_attn.v_proj.weight": "model-00010-of-00019.safetensors",
287
+ "model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00019.safetensors",
288
+ "model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00011-of-00019.safetensors",
289
+ "model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00019.safetensors",
290
+ "model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00011-of-00019.safetensors",
291
+ "model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00011-of-00019.safetensors",
292
+ "model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00011-of-00019.safetensors",
293
+ "model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00011-of-00019.safetensors",
294
+ "model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00011-of-00019.safetensors",
295
+ "model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00019.safetensors",
296
+ "model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00019.safetensors",
297
+ "model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00011-of-00019.safetensors",
298
+ "model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00019.safetensors",
299
+ "model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00011-of-00019.safetensors",
300
+ "model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00011-of-00019.safetensors",
301
+ "model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00011-of-00019.safetensors",
302
+ "model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00011-of-00019.safetensors",
303
+ "model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00011-of-00019.safetensors",
304
+ "model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00011-of-00019.safetensors",
305
+ "model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00011-of-00019.safetensors",
306
+ "model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00011-of-00019.safetensors",
307
+ "model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00011-of-00019.safetensors",
308
+ "model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00011-of-00019.safetensors",
309
+ "model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00011-of-00019.safetensors",
310
+ "model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00011-of-00019.safetensors",
311
+ "model.layers.17.block_sparse_moe.gate.weight": "model-00011-of-00019.safetensors",
312
+ "model.layers.17.input_layernorm.weight": "model-00011-of-00019.safetensors",
313
+ "model.layers.17.post_attention_layernorm.weight": "model-00011-of-00019.safetensors",
314
+ "model.layers.17.self_attn.k_proj.weight": "model-00011-of-00019.safetensors",
315
+ "model.layers.17.self_attn.o_proj.weight": "model-00011-of-00019.safetensors",
316
+ "model.layers.17.self_attn.q_proj.weight": "model-00011-of-00019.safetensors",
317
+ "model.layers.17.self_attn.v_proj.weight": "model-00011-of-00019.safetensors",
318
+ "model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00019.safetensors",
319
+ "model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00011-of-00019.safetensors",
320
+ "model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00019.safetensors",
321
+ "model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00011-of-00019.safetensors",
322
+ "model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00011-of-00019.safetensors",
323
+ "model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00011-of-00019.safetensors",
324
+ "model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00011-of-00019.safetensors",
325
+ "model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00011-of-00019.safetensors",
326
+ "model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00019.safetensors",
327
+ "model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00019.safetensors",
328
+ "model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00011-of-00019.safetensors",
329
+ "model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00019.safetensors",
330
+ "model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00011-of-00019.safetensors",
331
+ "model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00011-of-00019.safetensors",
332
+ "model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00011-of-00019.safetensors",
333
+ "model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00011-of-00019.safetensors",
334
+ "model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00012-of-00019.safetensors",
335
+ "model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00012-of-00019.safetensors",
336
+ "model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00012-of-00019.safetensors",
337
+ "model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00012-of-00019.safetensors",
338
+ "model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00012-of-00019.safetensors",
339
+ "model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00012-of-00019.safetensors",
340
+ "model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00012-of-00019.safetensors",
341
+ "model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00012-of-00019.safetensors",
342
+ "model.layers.18.block_sparse_moe.gate.weight": "model-00011-of-00019.safetensors",
343
+ "model.layers.18.input_layernorm.weight": "model-00012-of-00019.safetensors",
344
+ "model.layers.18.post_attention_layernorm.weight": "model-00012-of-00019.safetensors",
345
+ "model.layers.18.self_attn.k_proj.weight": "model-00011-of-00019.safetensors",
346
+ "model.layers.18.self_attn.o_proj.weight": "model-00011-of-00019.safetensors",
347
+ "model.layers.18.self_attn.q_proj.weight": "model-00011-of-00019.safetensors",
348
+ "model.layers.18.self_attn.v_proj.weight": "model-00011-of-00019.safetensors",
349
+ "model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00012-of-00019.safetensors",
350
+ "model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00012-of-00019.safetensors",
351
+ "model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00012-of-00019.safetensors",
352
+ "model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00012-of-00019.safetensors",
353
+ "model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00019.safetensors",
354
+ "model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00012-of-00019.safetensors",
355
+ "model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00012-of-00019.safetensors",
356
+ "model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00019.safetensors",
357
+ "model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00012-of-00019.safetensors",
358
+ "model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00012-of-00019.safetensors",
359
+ "model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00012-of-00019.safetensors",
360
+ "model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00012-of-00019.safetensors",
361
+ "model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00012-of-00019.safetensors",
362
+ "model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00012-of-00019.safetensors",
363
+ "model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00012-of-00019.safetensors",
364
+ "model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00012-of-00019.safetensors",
365
+ "model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00012-of-00019.safetensors",
366
+ "model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00012-of-00019.safetensors",
367
+ "model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00012-of-00019.safetensors",
368
+ "model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00012-of-00019.safetensors",
369
+ "model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00012-of-00019.safetensors",
370
+ "model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00012-of-00019.safetensors",
371
+ "model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00012-of-00019.safetensors",
372
+ "model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00012-of-00019.safetensors",
373
+ "model.layers.19.block_sparse_moe.gate.weight": "model-00012-of-00019.safetensors",
374
+ "model.layers.19.input_layernorm.weight": "model-00012-of-00019.safetensors",
375
+ "model.layers.19.post_attention_layernorm.weight": "model-00012-of-00019.safetensors",
376
+ "model.layers.19.self_attn.k_proj.weight": "model-00012-of-00019.safetensors",
377
+ "model.layers.19.self_attn.o_proj.weight": "model-00012-of-00019.safetensors",
378
+ "model.layers.19.self_attn.q_proj.weight": "model-00012-of-00019.safetensors",
379
+ "model.layers.19.self_attn.v_proj.weight": "model-00012-of-00019.safetensors",
380
+ "model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00019.safetensors",
381
+ "model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00019.safetensors",
382
+ "model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00019.safetensors",
383
+ "model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00019.safetensors",
384
+ "model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00019.safetensors",
385
+ "model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00019.safetensors",
386
+ "model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00019.safetensors",
387
+ "model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00019.safetensors",
388
+ "model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00019.safetensors",
389
+ "model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00019.safetensors",
390
+ "model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00019.safetensors",
391
+ "model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00019.safetensors",
392
+ "model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00019.safetensors",
393
+ "model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00019.safetensors",
394
+ "model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00019.safetensors",
395
+ "model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00019.safetensors",
396
+ "model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00019.safetensors",
397
+ "model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00019.safetensors",
398
+ "model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00019.safetensors",
399
+ "model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00019.safetensors",
400
+ "model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00019.safetensors",
401
+ "model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00019.safetensors",
402
+ "model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00019.safetensors",
403
+ "model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00019.safetensors",
404
+ "model.layers.2.block_sparse_moe.gate.weight": "model-00002-of-00019.safetensors",
405
+ "model.layers.2.input_layernorm.weight": "model-00002-of-00019.safetensors",
406
+ "model.layers.2.post_attention_layernorm.weight": "model-00002-of-00019.safetensors",
407
+ "model.layers.2.self_attn.k_proj.weight": "model-00002-of-00019.safetensors",
408
+ "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00019.safetensors",
409
+ "model.layers.2.self_attn.q_proj.weight": "model-00002-of-00019.safetensors",
410
+ "model.layers.2.self_attn.v_proj.weight": "model-00002-of-00019.safetensors",
411
+ "model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00012-of-00019.safetensors",
412
+ "model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00012-of-00019.safetensors",
413
+ "model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00012-of-00019.safetensors",
414
+ "model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00012-of-00019.safetensors",
415
+ "model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00019.safetensors",
416
+ "model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00012-of-00019.safetensors",
417
+ "model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00012-of-00019.safetensors",
418
+ "model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00019.safetensors",
419
+ "model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00012-of-00019.safetensors",
420
+ "model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00019.safetensors",
421
+ "model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00013-of-00019.safetensors",
422
+ "model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00019.safetensors",
423
+ "model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00013-of-00019.safetensors",
424
+ "model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00013-of-00019.safetensors",
425
+ "model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00013-of-00019.safetensors",
426
+ "model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00013-of-00019.safetensors",
427
+ "model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00013-of-00019.safetensors",
428
+ "model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00013-of-00019.safetensors",
429
+ "model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00013-of-00019.safetensors",
430
+ "model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00013-of-00019.safetensors",
431
+ "model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00013-of-00019.safetensors",
432
+ "model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00013-of-00019.safetensors",
433
+ "model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00013-of-00019.safetensors",
434
+ "model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00013-of-00019.safetensors",
435
+ "model.layers.20.block_sparse_moe.gate.weight": "model-00012-of-00019.safetensors",
436
+ "model.layers.20.input_layernorm.weight": "model-00013-of-00019.safetensors",
437
+ "model.layers.20.post_attention_layernorm.weight": "model-00013-of-00019.safetensors",
438
+ "model.layers.20.self_attn.k_proj.weight": "model-00012-of-00019.safetensors",
439
+ "model.layers.20.self_attn.o_proj.weight": "model-00012-of-00019.safetensors",
440
+ "model.layers.20.self_attn.q_proj.weight": "model-00012-of-00019.safetensors",
441
+ "model.layers.20.self_attn.v_proj.weight": "model-00012-of-00019.safetensors",
442
+ "model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00019.safetensors",
443
+ "model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00013-of-00019.safetensors",
444
+ "model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00013-of-00019.safetensors",
445
+ "model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00013-of-00019.safetensors",
446
+ "model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00013-of-00019.safetensors",
447
+ "model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00013-of-00019.safetensors",
448
+ "model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00013-of-00019.safetensors",
449
+ "model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00013-of-00019.safetensors",
450
+ "model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00013-of-00019.safetensors",
451
+ "model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00019.safetensors",
452
+ "model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00013-of-00019.safetensors",
453
+ "model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00019.safetensors",
454
+ "model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00013-of-00019.safetensors",
455
+ "model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00013-of-00019.safetensors",
456
+ "model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00013-of-00019.safetensors",
457
+ "model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00013-of-00019.safetensors",
458
+ "model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00013-of-00019.safetensors",
459
+ "model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00013-of-00019.safetensors",
460
+ "model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00013-of-00019.safetensors",
461
+ "model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00013-of-00019.safetensors",
462
+ "model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00013-of-00019.safetensors",
463
+ "model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00013-of-00019.safetensors",
464
+ "model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00013-of-00019.safetensors",
465
+ "model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00013-of-00019.safetensors",
466
+ "model.layers.21.block_sparse_moe.gate.weight": "model-00013-of-00019.safetensors",
467
+ "model.layers.21.input_layernorm.weight": "model-00013-of-00019.safetensors",
468
+ "model.layers.21.post_attention_layernorm.weight": "model-00013-of-00019.safetensors",
469
+ "model.layers.21.self_attn.k_proj.weight": "model-00013-of-00019.safetensors",
470
+ "model.layers.21.self_attn.o_proj.weight": "model-00013-of-00019.safetensors",
471
+ "model.layers.21.self_attn.q_proj.weight": "model-00013-of-00019.safetensors",
472
+ "model.layers.21.self_attn.v_proj.weight": "model-00013-of-00019.safetensors",
473
+ "model.layers.22.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00019.safetensors",
474
+ "model.layers.22.block_sparse_moe.experts.0.w2.weight": "model-00013-of-00019.safetensors",
475
+ "model.layers.22.block_sparse_moe.experts.0.w3.weight": "model-00014-of-00019.safetensors",
476
+ "model.layers.22.block_sparse_moe.experts.1.w1.weight": "model-00014-of-00019.safetensors",
477
+ "model.layers.22.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00019.safetensors",
478
+ "model.layers.22.block_sparse_moe.experts.1.w3.weight": "model-00014-of-00019.safetensors",
479
+ "model.layers.22.block_sparse_moe.experts.2.w1.weight": "model-00014-of-00019.safetensors",
480
+ "model.layers.22.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00019.safetensors",
481
+ "model.layers.22.block_sparse_moe.experts.2.w3.weight": "model-00014-of-00019.safetensors",
482
+ "model.layers.22.block_sparse_moe.experts.3.w1.weight": "model-00014-of-00019.safetensors",
483
+ "model.layers.22.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00019.safetensors",
484
+ "model.layers.22.block_sparse_moe.experts.3.w3.weight": "model-00014-of-00019.safetensors",
485
+ "model.layers.22.block_sparse_moe.experts.4.w1.weight": "model-00014-of-00019.safetensors",
486
+ "model.layers.22.block_sparse_moe.experts.4.w2.weight": "model-00014-of-00019.safetensors",
487
+ "model.layers.22.block_sparse_moe.experts.4.w3.weight": "model-00014-of-00019.safetensors",
488
+ "model.layers.22.block_sparse_moe.experts.5.w1.weight": "model-00014-of-00019.safetensors",
489
+ "model.layers.22.block_sparse_moe.experts.5.w2.weight": "model-00014-of-00019.safetensors",
490
+ "model.layers.22.block_sparse_moe.experts.5.w3.weight": "model-00014-of-00019.safetensors",
491
+ "model.layers.22.block_sparse_moe.experts.6.w1.weight": "model-00014-of-00019.safetensors",
492
+ "model.layers.22.block_sparse_moe.experts.6.w2.weight": "model-00014-of-00019.safetensors",
493
+ "model.layers.22.block_sparse_moe.experts.6.w3.weight": "model-00014-of-00019.safetensors",
494
+ "model.layers.22.block_sparse_moe.experts.7.w1.weight": "model-00014-of-00019.safetensors",
495
+ "model.layers.22.block_sparse_moe.experts.7.w2.weight": "model-00014-of-00019.safetensors",
496
+ "model.layers.22.block_sparse_moe.experts.7.w3.weight": "model-00014-of-00019.safetensors",
497
+ "model.layers.22.block_sparse_moe.gate.weight": "model-00013-of-00019.safetensors",
498
+ "model.layers.22.input_layernorm.weight": "model-00014-of-00019.safetensors",
499
+ "model.layers.22.post_attention_layernorm.weight": "model-00014-of-00019.safetensors",
500
+ "model.layers.22.self_attn.k_proj.weight": "model-00013-of-00019.safetensors",
501
+ "model.layers.22.self_attn.o_proj.weight": "model-00013-of-00019.safetensors",
502
+ "model.layers.22.self_attn.q_proj.weight": "model-00013-of-00019.safetensors",
503
+ "model.layers.22.self_attn.v_proj.weight": "model-00013-of-00019.safetensors",
504
+ "model.layers.23.block_sparse_moe.experts.0.w1.weight": "model-00014-of-00019.safetensors",
505
+ "model.layers.23.block_sparse_moe.experts.0.w2.weight": "model-00014-of-00019.safetensors",
506
+ "model.layers.23.block_sparse_moe.experts.0.w3.weight": "model-00014-of-00019.safetensors",
507
+ "model.layers.23.block_sparse_moe.experts.1.w1.weight": "model-00014-of-00019.safetensors",
508
+ "model.layers.23.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00019.safetensors",
509
+ "model.layers.23.block_sparse_moe.experts.1.w3.weight": "model-00014-of-00019.safetensors",
510
+ "model.layers.23.block_sparse_moe.experts.2.w1.weight": "model-00014-of-00019.safetensors",
511
+ "model.layers.23.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00019.safetensors",
512
+ "model.layers.23.block_sparse_moe.experts.2.w3.weight": "model-00014-of-00019.safetensors",
513
+ "model.layers.23.block_sparse_moe.experts.3.w1.weight": "model-00014-of-00019.safetensors",
514
+ "model.layers.23.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00019.safetensors",
515
+ "model.layers.23.block_sparse_moe.experts.3.w3.weight": "model-00014-of-00019.safetensors",
516
+ "model.layers.23.block_sparse_moe.experts.4.w1.weight": "model-00014-of-00019.safetensors",
517
+ "model.layers.23.block_sparse_moe.experts.4.w2.weight": "model-00014-of-00019.safetensors",
518
+ "model.layers.23.block_sparse_moe.experts.4.w3.weight": "model-00014-of-00019.safetensors",
519
+ "model.layers.23.block_sparse_moe.experts.5.w1.weight": "model-00014-of-00019.safetensors",
520
+ "model.layers.23.block_sparse_moe.experts.5.w2.weight": "model-00014-of-00019.safetensors",
521
+ "model.layers.23.block_sparse_moe.experts.5.w3.weight": "model-00014-of-00019.safetensors",
522
+ "model.layers.23.block_sparse_moe.experts.6.w1.weight": "model-00014-of-00019.safetensors",
523
+ "model.layers.23.block_sparse_moe.experts.6.w2.weight": "model-00015-of-00019.safetensors",
524
+ "model.layers.23.block_sparse_moe.experts.6.w3.weight": "model-00015-of-00019.safetensors",
525
+ "model.layers.23.block_sparse_moe.experts.7.w1.weight": "model-00015-of-00019.safetensors",
526
+ "model.layers.23.block_sparse_moe.experts.7.w2.weight": "model-00015-of-00019.safetensors",
527
+ "model.layers.23.block_sparse_moe.experts.7.w3.weight": "model-00015-of-00019.safetensors",
528
+ "model.layers.23.block_sparse_moe.gate.weight": "model-00014-of-00019.safetensors",
529
+ "model.layers.23.input_layernorm.weight": "model-00015-of-00019.safetensors",
530
+ "model.layers.23.post_attention_layernorm.weight": "model-00015-of-00019.safetensors",
531
+ "model.layers.23.self_attn.k_proj.weight": "model-00014-of-00019.safetensors",
532
+ "model.layers.23.self_attn.o_proj.weight": "model-00014-of-00019.safetensors",
533
+ "model.layers.23.self_attn.q_proj.weight": "model-00014-of-00019.safetensors",
534
+ "model.layers.23.self_attn.v_proj.weight": "model-00014-of-00019.safetensors",
535
+ "model.layers.24.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00019.safetensors",
536
+ "model.layers.24.block_sparse_moe.experts.0.w2.weight": "model-00015-of-00019.safetensors",
537
+ "model.layers.24.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00019.safetensors",
538
+ "model.layers.24.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00019.safetensors",
539
+ "model.layers.24.block_sparse_moe.experts.1.w2.weight": "model-00015-of-00019.safetensors",
540
+ "model.layers.24.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00019.safetensors",
541
+ "model.layers.24.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00019.safetensors",
542
+ "model.layers.24.block_sparse_moe.experts.2.w2.weight": "model-00015-of-00019.safetensors",
543
+ "model.layers.24.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00019.safetensors",
544
+ "model.layers.24.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00019.safetensors",
545
+ "model.layers.24.block_sparse_moe.experts.3.w2.weight": "model-00015-of-00019.safetensors",
546
+ "model.layers.24.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00019.safetensors",
547
+ "model.layers.24.block_sparse_moe.experts.4.w1.weight": "model-00015-of-00019.safetensors",
548
+ "model.layers.24.block_sparse_moe.experts.4.w2.weight": "model-00015-of-00019.safetensors",
549
+ "model.layers.24.block_sparse_moe.experts.4.w3.weight": "model-00015-of-00019.safetensors",
550
+ "model.layers.24.block_sparse_moe.experts.5.w1.weight": "model-00015-of-00019.safetensors",
551
+ "model.layers.24.block_sparse_moe.experts.5.w2.weight": "model-00015-of-00019.safetensors",
552
+ "model.layers.24.block_sparse_moe.experts.5.w3.weight": "model-00015-of-00019.safetensors",
553
+ "model.layers.24.block_sparse_moe.experts.6.w1.weight": "model-00015-of-00019.safetensors",
554
+ "model.layers.24.block_sparse_moe.experts.6.w2.weight": "model-00015-of-00019.safetensors",
555
+ "model.layers.24.block_sparse_moe.experts.6.w3.weight": "model-00015-of-00019.safetensors",
556
+ "model.layers.24.block_sparse_moe.experts.7.w1.weight": "model-00015-of-00019.safetensors",
557
+ "model.layers.24.block_sparse_moe.experts.7.w2.weight": "model-00015-of-00019.safetensors",
558
+ "model.layers.24.block_sparse_moe.experts.7.w3.weight": "model-00015-of-00019.safetensors",
559
+ "model.layers.24.block_sparse_moe.gate.weight": "model-00015-of-00019.safetensors",
560
+ "model.layers.24.input_layernorm.weight": "model-00015-of-00019.safetensors",
561
+ "model.layers.24.post_attention_layernorm.weight": "model-00015-of-00019.safetensors",
562
+ "model.layers.24.self_attn.k_proj.weight": "model-00015-of-00019.safetensors",
563
+ "model.layers.24.self_attn.o_proj.weight": "model-00015-of-00019.safetensors",
564
+ "model.layers.24.self_attn.q_proj.weight": "model-00015-of-00019.safetensors",
565
+ "model.layers.24.self_attn.v_proj.weight": "model-00015-of-00019.safetensors",
566
+ "model.layers.25.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00019.safetensors",
567
+ "model.layers.25.block_sparse_moe.experts.0.w2.weight": "model-00015-of-00019.safetensors",
568
+ "model.layers.25.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00019.safetensors",
569
+ "model.layers.25.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00019.safetensors",
570
+ "model.layers.25.block_sparse_moe.experts.1.w2.weight": "model-00015-of-00019.safetensors",
571
+ "model.layers.25.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00019.safetensors",
572
+ "model.layers.25.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00019.safetensors",
573
+ "model.layers.25.block_sparse_moe.experts.2.w2.weight": "model-00015-of-00019.safetensors",
574
+ "model.layers.25.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00019.safetensors",
575
+ "model.layers.25.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00019.safetensors",
576
+ "model.layers.25.block_sparse_moe.experts.3.w2.weight": "model-00015-of-00019.safetensors",
577
+ "model.layers.25.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00019.safetensors",
578
+ "model.layers.25.block_sparse_moe.experts.4.w1.weight": "model-00016-of-00019.safetensors",
579
+ "model.layers.25.block_sparse_moe.experts.4.w2.weight": "model-00016-of-00019.safetensors",
580
+ "model.layers.25.block_sparse_moe.experts.4.w3.weight": "model-00016-of-00019.safetensors",
581
+ "model.layers.25.block_sparse_moe.experts.5.w1.weight": "model-00016-of-00019.safetensors",
582
+ "model.layers.25.block_sparse_moe.experts.5.w2.weight": "model-00016-of-00019.safetensors",
583
+ "model.layers.25.block_sparse_moe.experts.5.w3.weight": "model-00016-of-00019.safetensors",
584
+ "model.layers.25.block_sparse_moe.experts.6.w1.weight": "model-00016-of-00019.safetensors",
585
+ "model.layers.25.block_sparse_moe.experts.6.w2.weight": "model-00016-of-00019.safetensors",
586
+ "model.layers.25.block_sparse_moe.experts.6.w3.weight": "model-00016-of-00019.safetensors",
587
+ "model.layers.25.block_sparse_moe.experts.7.w1.weight": "model-00016-of-00019.safetensors",
588
+ "model.layers.25.block_sparse_moe.experts.7.w2.weight": "model-00016-of-00019.safetensors",
589
+ "model.layers.25.block_sparse_moe.experts.7.w3.weight": "model-00016-of-00019.safetensors",
590
+ "model.layers.25.block_sparse_moe.gate.weight": "model-00015-of-00019.safetensors",
591
+ "model.layers.25.input_layernorm.weight": "model-00016-of-00019.safetensors",
592
+ "model.layers.25.post_attention_layernorm.weight": "model-00016-of-00019.safetensors",
593
+ "model.layers.25.self_attn.k_proj.weight": "model-00015-of-00019.safetensors",
594
+ "model.layers.25.self_attn.o_proj.weight": "model-00015-of-00019.safetensors",
595
+ "model.layers.25.self_attn.q_proj.weight": "model-00015-of-00019.safetensors",
596
+ "model.layers.25.self_attn.v_proj.weight": "model-00015-of-00019.safetensors",
597
+ "model.layers.26.block_sparse_moe.experts.0.w1.weight": "model-00016-of-00019.safetensors",
598
+ "model.layers.26.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00019.safetensors",
599
+ "model.layers.26.block_sparse_moe.experts.0.w3.weight": "model-00016-of-00019.safetensors",
600
+ "model.layers.26.block_sparse_moe.experts.1.w1.weight": "model-00016-of-00019.safetensors",
601
+ "model.layers.26.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00019.safetensors",
602
+ "model.layers.26.block_sparse_moe.experts.1.w3.weight": "model-00016-of-00019.safetensors",
603
+ "model.layers.26.block_sparse_moe.experts.2.w1.weight": "model-00016-of-00019.safetensors",
604
+ "model.layers.26.block_sparse_moe.experts.2.w2.weight": "model-00016-of-00019.safetensors",
605
+ "model.layers.26.block_sparse_moe.experts.2.w3.weight": "model-00016-of-00019.safetensors",
606
+ "model.layers.26.block_sparse_moe.experts.3.w1.weight": "model-00016-of-00019.safetensors",
607
+ "model.layers.26.block_sparse_moe.experts.3.w2.weight": "model-00016-of-00019.safetensors",
608
+ "model.layers.26.block_sparse_moe.experts.3.w3.weight": "model-00016-of-00019.safetensors",
609
+ "model.layers.26.block_sparse_moe.experts.4.w1.weight": "model-00016-of-00019.safetensors",
610
+ "model.layers.26.block_sparse_moe.experts.4.w2.weight": "model-00016-of-00019.safetensors",
611
+ "model.layers.26.block_sparse_moe.experts.4.w3.weight": "model-00016-of-00019.safetensors",
612
+ "model.layers.26.block_sparse_moe.experts.5.w1.weight": "model-00016-of-00019.safetensors",
613
+ "model.layers.26.block_sparse_moe.experts.5.w2.weight": "model-00016-of-00019.safetensors",
614
+ "model.layers.26.block_sparse_moe.experts.5.w3.weight": "model-00016-of-00019.safetensors",
615
+ "model.layers.26.block_sparse_moe.experts.6.w1.weight": "model-00016-of-00019.safetensors",
616
+ "model.layers.26.block_sparse_moe.experts.6.w2.weight": "model-00016-of-00019.safetensors",
617
+ "model.layers.26.block_sparse_moe.experts.6.w3.weight": "model-00016-of-00019.safetensors",
618
+ "model.layers.26.block_sparse_moe.experts.7.w1.weight": "model-00016-of-00019.safetensors",
619
+ "model.layers.26.block_sparse_moe.experts.7.w2.weight": "model-00016-of-00019.safetensors",
620
+ "model.layers.26.block_sparse_moe.experts.7.w3.weight": "model-00016-of-00019.safetensors",
621
+ "model.layers.26.block_sparse_moe.gate.weight": "model-00016-of-00019.safetensors",
622
+ "model.layers.26.input_layernorm.weight": "model-00016-of-00019.safetensors",
623
+ "model.layers.26.post_attention_layernorm.weight": "model-00016-of-00019.safetensors",
624
+ "model.layers.26.self_attn.k_proj.weight": "model-00016-of-00019.safetensors",
625
+ "model.layers.26.self_attn.o_proj.weight": "model-00016-of-00019.safetensors",
626
+ "model.layers.26.self_attn.q_proj.weight": "model-00016-of-00019.safetensors",
627
+ "model.layers.26.self_attn.v_proj.weight": "model-00016-of-00019.safetensors",
628
+ "model.layers.27.block_sparse_moe.experts.0.w1.weight": "model-00016-of-00019.safetensors",
629
+ "model.layers.27.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00019.safetensors",
630
+ "model.layers.27.block_sparse_moe.experts.0.w3.weight": "model-00016-of-00019.safetensors",
631
+ "model.layers.27.block_sparse_moe.experts.1.w1.weight": "model-00016-of-00019.safetensors",
632
+ "model.layers.27.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00019.safetensors",
633
+ "model.layers.27.block_sparse_moe.experts.1.w3.weight": "model-00017-of-00019.safetensors",
634
+ "model.layers.27.block_sparse_moe.experts.2.w1.weight": "model-00017-of-00019.safetensors",
635
+ "model.layers.27.block_sparse_moe.experts.2.w2.weight": "model-00017-of-00019.safetensors",
636
+ "model.layers.27.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00019.safetensors",
637
+ "model.layers.27.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00019.safetensors",
638
+ "model.layers.27.block_sparse_moe.experts.3.w2.weight": "model-00017-of-00019.safetensors",
639
+ "model.layers.27.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00019.safetensors",
640
+ "model.layers.27.block_sparse_moe.experts.4.w1.weight": "model-00017-of-00019.safetensors",
641
+ "model.layers.27.block_sparse_moe.experts.4.w2.weight": "model-00017-of-00019.safetensors",
642
+ "model.layers.27.block_sparse_moe.experts.4.w3.weight": "model-00017-of-00019.safetensors",
643
+ "model.layers.27.block_sparse_moe.experts.5.w1.weight": "model-00017-of-00019.safetensors",
644
+ "model.layers.27.block_sparse_moe.experts.5.w2.weight": "model-00017-of-00019.safetensors",
645
+ "model.layers.27.block_sparse_moe.experts.5.w3.weight": "model-00017-of-00019.safetensors",
646
+ "model.layers.27.block_sparse_moe.experts.6.w1.weight": "model-00017-of-00019.safetensors",
647
+ "model.layers.27.block_sparse_moe.experts.6.w2.weight": "model-00017-of-00019.safetensors",
648
+ "model.layers.27.block_sparse_moe.experts.6.w3.weight": "model-00017-of-00019.safetensors",
649
+ "model.layers.27.block_sparse_moe.experts.7.w1.weight": "model-00017-of-00019.safetensors",
650
+ "model.layers.27.block_sparse_moe.experts.7.w2.weight": "model-00017-of-00019.safetensors",
651
+ "model.layers.27.block_sparse_moe.experts.7.w3.weight": "model-00017-of-00019.safetensors",
652
+ "model.layers.27.block_sparse_moe.gate.weight": "model-00016-of-00019.safetensors",
653
+ "model.layers.27.input_layernorm.weight": "model-00017-of-00019.safetensors",
654
+ "model.layers.27.post_attention_layernorm.weight": "model-00017-of-00019.safetensors",
655
+ "model.layers.27.self_attn.k_proj.weight": "model-00016-of-00019.safetensors",
656
+ "model.layers.27.self_attn.o_proj.weight": "model-00016-of-00019.safetensors",
657
+ "model.layers.27.self_attn.q_proj.weight": "model-00016-of-00019.safetensors",
658
+ "model.layers.27.self_attn.v_proj.weight": "model-00016-of-00019.safetensors",
659
+ "model.layers.28.block_sparse_moe.experts.0.w1.weight": "model-00017-of-00019.safetensors",
660
+ "model.layers.28.block_sparse_moe.experts.0.w2.weight": "model-00017-of-00019.safetensors",
661
+ "model.layers.28.block_sparse_moe.experts.0.w3.weight": "model-00017-of-00019.safetensors",
662
+ "model.layers.28.block_sparse_moe.experts.1.w1.weight": "model-00017-of-00019.safetensors",
663
+ "model.layers.28.block_sparse_moe.experts.1.w2.weight": "model-00017-of-00019.safetensors",
664
+ "model.layers.28.block_sparse_moe.experts.1.w3.weight": "model-00017-of-00019.safetensors",
665
+ "model.layers.28.block_sparse_moe.experts.2.w1.weight": "model-00017-of-00019.safetensors",
666
+ "model.layers.28.block_sparse_moe.experts.2.w2.weight": "model-00017-of-00019.safetensors",
667
+ "model.layers.28.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00019.safetensors",
668
+ "model.layers.28.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00019.safetensors",
669
+ "model.layers.28.block_sparse_moe.experts.3.w2.weight": "model-00017-of-00019.safetensors",
670
+ "model.layers.28.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00019.safetensors",
671
+ "model.layers.28.block_sparse_moe.experts.4.w1.weight": "model-00017-of-00019.safetensors",
672
+ "model.layers.28.block_sparse_moe.experts.4.w2.weight": "model-00017-of-00019.safetensors",
673
+ "model.layers.28.block_sparse_moe.experts.4.w3.weight": "model-00017-of-00019.safetensors",
674
+ "model.layers.28.block_sparse_moe.experts.5.w1.weight": "model-00017-of-00019.safetensors",
675
+ "model.layers.28.block_sparse_moe.experts.5.w2.weight": "model-00017-of-00019.safetensors",
676
+ "model.layers.28.block_sparse_moe.experts.5.w3.weight": "model-00017-of-00019.safetensors",
677
+ "model.layers.28.block_sparse_moe.experts.6.w1.weight": "model-00017-of-00019.safetensors",
678
+ "model.layers.28.block_sparse_moe.experts.6.w2.weight": "model-00017-of-00019.safetensors",
679
+ "model.layers.28.block_sparse_moe.experts.6.w3.weight": "model-00017-of-00019.safetensors",
680
+ "model.layers.28.block_sparse_moe.experts.7.w1.weight": "model-00017-of-00019.safetensors",
681
+ "model.layers.28.block_sparse_moe.experts.7.w2.weight": "model-00018-of-00019.safetensors",
682
+ "model.layers.28.block_sparse_moe.experts.7.w3.weight": "model-00018-of-00019.safetensors",
683
+ "model.layers.28.block_sparse_moe.gate.weight": "model-00017-of-00019.safetensors",
684
+ "model.layers.28.input_layernorm.weight": "model-00018-of-00019.safetensors",
685
+ "model.layers.28.post_attention_layernorm.weight": "model-00018-of-00019.safetensors",
686
+ "model.layers.28.self_attn.k_proj.weight": "model-00017-of-00019.safetensors",
687
+ "model.layers.28.self_attn.o_proj.weight": "model-00017-of-00019.safetensors",
688
+ "model.layers.28.self_attn.q_proj.weight": "model-00017-of-00019.safetensors",
689
+ "model.layers.28.self_attn.v_proj.weight": "model-00017-of-00019.safetensors",
690
+ "model.layers.29.block_sparse_moe.experts.0.w1.weight": "model-00018-of-00019.safetensors",
691
+ "model.layers.29.block_sparse_moe.experts.0.w2.weight": "model-00018-of-00019.safetensors",
692
+ "model.layers.29.block_sparse_moe.experts.0.w3.weight": "model-00018-of-00019.safetensors",
693
+ "model.layers.29.block_sparse_moe.experts.1.w1.weight": "model-00018-of-00019.safetensors",
694
+ "model.layers.29.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00019.safetensors",
695
+ "model.layers.29.block_sparse_moe.experts.1.w3.weight": "model-00018-of-00019.safetensors",
696
+ "model.layers.29.block_sparse_moe.experts.2.w1.weight": "model-00018-of-00019.safetensors",
697
+ "model.layers.29.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00019.safetensors",
698
+ "model.layers.29.block_sparse_moe.experts.2.w3.weight": "model-00018-of-00019.safetensors",
699
+ "model.layers.29.block_sparse_moe.experts.3.w1.weight": "model-00018-of-00019.safetensors",
700
+ "model.layers.29.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00019.safetensors",
701
+ "model.layers.29.block_sparse_moe.experts.3.w3.weight": "model-00018-of-00019.safetensors",
702
+ "model.layers.29.block_sparse_moe.experts.4.w1.weight": "model-00018-of-00019.safetensors",
703
+ "model.layers.29.block_sparse_moe.experts.4.w2.weight": "model-00018-of-00019.safetensors",
704
+ "model.layers.29.block_sparse_moe.experts.4.w3.weight": "model-00018-of-00019.safetensors",
705
+ "model.layers.29.block_sparse_moe.experts.5.w1.weight": "model-00018-of-00019.safetensors",
706
+ "model.layers.29.block_sparse_moe.experts.5.w2.weight": "model-00018-of-00019.safetensors",
707
+ "model.layers.29.block_sparse_moe.experts.5.w3.weight": "model-00018-of-00019.safetensors",
708
+ "model.layers.29.block_sparse_moe.experts.6.w1.weight": "model-00018-of-00019.safetensors",
709
+ "model.layers.29.block_sparse_moe.experts.6.w2.weight": "model-00018-of-00019.safetensors",
710
+ "model.layers.29.block_sparse_moe.experts.6.w3.weight": "model-00018-of-00019.safetensors",
711
+ "model.layers.29.block_sparse_moe.experts.7.w1.weight": "model-00018-of-00019.safetensors",
712
+ "model.layers.29.block_sparse_moe.experts.7.w2.weight": "model-00018-of-00019.safetensors",
713
+ "model.layers.29.block_sparse_moe.experts.7.w3.weight": "model-00018-of-00019.safetensors",
714
+ "model.layers.29.block_sparse_moe.gate.weight": "model-00018-of-00019.safetensors",
715
+ "model.layers.29.input_layernorm.weight": "model-00018-of-00019.safetensors",
716
+ "model.layers.29.post_attention_layernorm.weight": "model-00018-of-00019.safetensors",
717
+ "model.layers.29.self_attn.k_proj.weight": "model-00018-of-00019.safetensors",
718
+ "model.layers.29.self_attn.o_proj.weight": "model-00018-of-00019.safetensors",
719
+ "model.layers.29.self_attn.q_proj.weight": "model-00018-of-00019.safetensors",
720
+ "model.layers.29.self_attn.v_proj.weight": "model-00018-of-00019.safetensors",
721
+ "model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00019.safetensors",
722
+ "model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00019.safetensors",
723
+ "model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00019.safetensors",
724
+ "model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00019.safetensors",
725
+ "model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00019.safetensors",
726
+ "model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00019.safetensors",
727
+ "model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00019.safetensors",
728
+ "model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00019.safetensors",
729
+ "model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00019.safetensors",
730
+ "model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00019.safetensors",
731
+ "model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00019.safetensors",
732
+ "model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00019.safetensors",
733
+ "model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00019.safetensors",
734
+ "model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00019.safetensors",
735
+ "model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00019.safetensors",
736
+ "model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00019.safetensors",
737
+ "model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00019.safetensors",
738
+ "model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00019.safetensors",
739
+ "model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00019.safetensors",
740
+ "model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00019.safetensors",
741
+ "model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00019.safetensors",
742
+ "model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00019.safetensors",
743
+ "model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00019.safetensors",
744
+ "model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00019.safetensors",
745
+ "model.layers.3.block_sparse_moe.gate.weight": "model-00002-of-00019.safetensors",
746
+ "model.layers.3.input_layernorm.weight": "model-00003-of-00019.safetensors",
747
+ "model.layers.3.post_attention_layernorm.weight": "model-00003-of-00019.safetensors",
748
+ "model.layers.3.self_attn.k_proj.weight": "model-00002-of-00019.safetensors",
749
+ "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00019.safetensors",
750
+ "model.layers.3.self_attn.q_proj.weight": "model-00002-of-00019.safetensors",
751
+ "model.layers.3.self_attn.v_proj.weight": "model-00002-of-00019.safetensors",
752
+ "model.layers.30.block_sparse_moe.experts.0.w1.weight": "model-00018-of-00019.safetensors",
753
+ "model.layers.30.block_sparse_moe.experts.0.w2.weight": "model-00018-of-00019.safetensors",
754
+ "model.layers.30.block_sparse_moe.experts.0.w3.weight": "model-00018-of-00019.safetensors",
755
+ "model.layers.30.block_sparse_moe.experts.1.w1.weight": "model-00018-of-00019.safetensors",
756
+ "model.layers.30.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00019.safetensors",
757
+ "model.layers.30.block_sparse_moe.experts.1.w3.weight": "model-00018-of-00019.safetensors",
758
+ "model.layers.30.block_sparse_moe.experts.2.w1.weight": "model-00018-of-00019.safetensors",
759
+ "model.layers.30.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00019.safetensors",
760
+ "model.layers.30.block_sparse_moe.experts.2.w3.weight": "model-00018-of-00019.safetensors",
761
+ "model.layers.30.block_sparse_moe.experts.3.w1.weight": "model-00018-of-00019.safetensors",
762
+ "model.layers.30.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00019.safetensors",
763
+ "model.layers.30.block_sparse_moe.experts.3.w3.weight": "model-00018-of-00019.safetensors",
764
+ "model.layers.30.block_sparse_moe.experts.4.w1.weight": "model-00018-of-00019.safetensors",
765
+ "model.layers.30.block_sparse_moe.experts.4.w2.weight": "model-00018-of-00019.safetensors",
766
+ "model.layers.30.block_sparse_moe.experts.4.w3.weight": "model-00018-of-00019.safetensors",
767
+ "model.layers.30.block_sparse_moe.experts.5.w1.weight": "model-00019-of-00019.safetensors",
768
+ "model.layers.30.block_sparse_moe.experts.5.w2.weight": "model-00019-of-00019.safetensors",
769
+ "model.layers.30.block_sparse_moe.experts.5.w3.weight": "model-00019-of-00019.safetensors",
770
+ "model.layers.30.block_sparse_moe.experts.6.w1.weight": "model-00019-of-00019.safetensors",
771
+ "model.layers.30.block_sparse_moe.experts.6.w2.weight": "model-00019-of-00019.safetensors",
772
+ "model.layers.30.block_sparse_moe.experts.6.w3.weight": "model-00019-of-00019.safetensors",
773
+ "model.layers.30.block_sparse_moe.experts.7.w1.weight": "model-00019-of-00019.safetensors",
774
+ "model.layers.30.block_sparse_moe.experts.7.w2.weight": "model-00019-of-00019.safetensors",
775
+ "model.layers.30.block_sparse_moe.experts.7.w3.weight": "model-00019-of-00019.safetensors",
776
+ "model.layers.30.block_sparse_moe.gate.weight": "model-00018-of-00019.safetensors",
777
+ "model.layers.30.input_layernorm.weight": "model-00019-of-00019.safetensors",
778
+ "model.layers.30.post_attention_layernorm.weight": "model-00019-of-00019.safetensors",
779
+ "model.layers.30.self_attn.k_proj.weight": "model-00018-of-00019.safetensors",
780
+ "model.layers.30.self_attn.o_proj.weight": "model-00018-of-00019.safetensors",
781
+ "model.layers.30.self_attn.q_proj.weight": "model-00018-of-00019.safetensors",
782
+ "model.layers.30.self_attn.v_proj.weight": "model-00018-of-00019.safetensors",
783
+ "model.layers.31.block_sparse_moe.experts.0.w1.weight": "model-00019-of-00019.safetensors",
784
+ "model.layers.31.block_sparse_moe.experts.0.w2.weight": "model-00019-of-00019.safetensors",
785
+ "model.layers.31.block_sparse_moe.experts.0.w3.weight": "model-00019-of-00019.safetensors",
786
+ "model.layers.31.block_sparse_moe.experts.1.w1.weight": "model-00019-of-00019.safetensors",
787
+ "model.layers.31.block_sparse_moe.experts.1.w2.weight": "model-00019-of-00019.safetensors",
788
+ "model.layers.31.block_sparse_moe.experts.1.w3.weight": "model-00019-of-00019.safetensors",
789
+ "model.layers.31.block_sparse_moe.experts.2.w1.weight": "model-00019-of-00019.safetensors",
790
+ "model.layers.31.block_sparse_moe.experts.2.w2.weight": "model-00019-of-00019.safetensors",
791
+ "model.layers.31.block_sparse_moe.experts.2.w3.weight": "model-00019-of-00019.safetensors",
792
+ "model.layers.31.block_sparse_moe.experts.3.w1.weight": "model-00019-of-00019.safetensors",
793
+ "model.layers.31.block_sparse_moe.experts.3.w2.weight": "model-00019-of-00019.safetensors",
794
+ "model.layers.31.block_sparse_moe.experts.3.w3.weight": "model-00019-of-00019.safetensors",
795
+ "model.layers.31.block_sparse_moe.experts.4.w1.weight": "model-00019-of-00019.safetensors",
796
+ "model.layers.31.block_sparse_moe.experts.4.w2.weight": "model-00019-of-00019.safetensors",
797
+ "model.layers.31.block_sparse_moe.experts.4.w3.weight": "model-00019-of-00019.safetensors",
798
+ "model.layers.31.block_sparse_moe.experts.5.w1.weight": "model-00019-of-00019.safetensors",
799
+ "model.layers.31.block_sparse_moe.experts.5.w2.weight": "model-00019-of-00019.safetensors",
800
+ "model.layers.31.block_sparse_moe.experts.5.w3.weight": "model-00019-of-00019.safetensors",
801
+ "model.layers.31.block_sparse_moe.experts.6.w1.weight": "model-00019-of-00019.safetensors",
802
+ "model.layers.31.block_sparse_moe.experts.6.w2.weight": "model-00019-of-00019.safetensors",
803
+ "model.layers.31.block_sparse_moe.experts.6.w3.weight": "model-00019-of-00019.safetensors",
804
+ "model.layers.31.block_sparse_moe.experts.7.w1.weight": "model-00019-of-00019.safetensors",
805
+ "model.layers.31.block_sparse_moe.experts.7.w2.weight": "model-00019-of-00019.safetensors",
806
+ "model.layers.31.block_sparse_moe.experts.7.w3.weight": "model-00019-of-00019.safetensors",
807
+ "model.layers.31.block_sparse_moe.gate.weight": "model-00019-of-00019.safetensors",
808
+ "model.layers.31.input_layernorm.weight": "model-00019-of-00019.safetensors",
809
+ "model.layers.31.post_attention_layernorm.weight": "model-00019-of-00019.safetensors",
810
+ "model.layers.31.self_attn.k_proj.weight": "model-00019-of-00019.safetensors",
811
+ "model.layers.31.self_attn.o_proj.weight": "model-00019-of-00019.safetensors",
812
+ "model.layers.31.self_attn.q_proj.weight": "model-00019-of-00019.safetensors",
813
+ "model.layers.31.self_attn.v_proj.weight": "model-00019-of-00019.safetensors",
814
+ "model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00019.safetensors",
815
+ "model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00019.safetensors",
816
+ "model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00019.safetensors",
817
+ "model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00019.safetensors",
818
+ "model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00019.safetensors",
819
+ "model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00019.safetensors",
820
+ "model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00019.safetensors",
821
+ "model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00019.safetensors",
822
+ "model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00019.safetensors",
823
+ "model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00019.safetensors",
824
+ "model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00019.safetensors",
825
+ "model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00019.safetensors",
826
+ "model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00019.safetensors",
827
+ "model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00019.safetensors",
828
+ "model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00019.safetensors",
829
+ "model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00019.safetensors",
830
+ "model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00019.safetensors",
831
+ "model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00019.safetensors",
832
+ "model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00019.safetensors",
833
+ "model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00019.safetensors",
834
+ "model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00019.safetensors",
835
+ "model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00019.safetensors",
836
+ "model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00019.safetensors",
837
+ "model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00019.safetensors",
838
+ "model.layers.4.block_sparse_moe.gate.weight": "model-00003-of-00019.safetensors",
839
+ "model.layers.4.input_layernorm.weight": "model-00003-of-00019.safetensors",
840
+ "model.layers.4.post_attention_layernorm.weight": "model-00003-of-00019.safetensors",
841
+ "model.layers.4.self_attn.k_proj.weight": "model-00003-of-00019.safetensors",
842
+ "model.layers.4.self_attn.o_proj.weight": "model-00003-of-00019.safetensors",
843
+ "model.layers.4.self_attn.q_proj.weight": "model-00003-of-00019.safetensors",
844
+ "model.layers.4.self_attn.v_proj.weight": "model-00003-of-00019.safetensors",
845
+ "model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00019.safetensors",
846
+ "model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00019.safetensors",
847
+ "model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00019.safetensors",
848
+ "model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00019.safetensors",
849
+ "model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00019.safetensors",
850
+ "model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00019.safetensors",
851
+ "model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00019.safetensors",
852
+ "model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00019.safetensors",
853
+ "model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00019.safetensors",
854
+ "model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00019.safetensors",
855
+ "model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00019.safetensors",
856
+ "model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00019.safetensors",
857
+ "model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00019.safetensors",
858
+ "model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00019.safetensors",
859
+ "model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00019.safetensors",
860
+ "model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00019.safetensors",
861
+ "model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00019.safetensors",
862
+ "model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00019.safetensors",
863
+ "model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00019.safetensors",
864
+ "model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00019.safetensors",
865
+ "model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00019.safetensors",
866
+ "model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00019.safetensors",
867
+ "model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00019.safetensors",
868
+ "model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00019.safetensors",
869
+ "model.layers.5.block_sparse_moe.gate.weight": "model-00003-of-00019.safetensors",
870
+ "model.layers.5.input_layernorm.weight": "model-00004-of-00019.safetensors",
871
+ "model.layers.5.post_attention_layernorm.weight": "model-00004-of-00019.safetensors",
872
+ "model.layers.5.self_attn.k_proj.weight": "model-00003-of-00019.safetensors",
873
+ "model.layers.5.self_attn.o_proj.weight": "model-00003-of-00019.safetensors",
874
+ "model.layers.5.self_attn.q_proj.weight": "model-00003-of-00019.safetensors",
875
+ "model.layers.5.self_attn.v_proj.weight": "model-00003-of-00019.safetensors",
876
+ "model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00019.safetensors",
877
+ "model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00019.safetensors",
878
+ "model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00019.safetensors",
879
+ "model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00019.safetensors",
880
+ "model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00019.safetensors",
881
+ "model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00019.safetensors",
882
+ "model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00019.safetensors",
883
+ "model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00019.safetensors",
884
+ "model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00019.safetensors",
885
+ "model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00019.safetensors",
886
+ "model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00019.safetensors",
887
+ "model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00019.safetensors",
888
+ "model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00019.safetensors",
889
+ "model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00019.safetensors",
890
+ "model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00019.safetensors",
891
+ "model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00019.safetensors",
892
+ "model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00019.safetensors",
893
+ "model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00019.safetensors",
894
+ "model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00019.safetensors",
895
+ "model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00019.safetensors",
896
+ "model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00019.safetensors",
897
+ "model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00019.safetensors",
898
+ "model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00019.safetensors",
899
+ "model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00019.safetensors",
900
+ "model.layers.6.block_sparse_moe.gate.weight": "model-00004-of-00019.safetensors",
901
+ "model.layers.6.input_layernorm.weight": "model-00005-of-00019.safetensors",
902
+ "model.layers.6.post_attention_layernorm.weight": "model-00005-of-00019.safetensors",
903
+ "model.layers.6.self_attn.k_proj.weight": "model-00004-of-00019.safetensors",
904
+ "model.layers.6.self_attn.o_proj.weight": "model-00004-of-00019.safetensors",
905
+ "model.layers.6.self_attn.q_proj.weight": "model-00004-of-00019.safetensors",
906
+ "model.layers.6.self_attn.v_proj.weight": "model-00004-of-00019.safetensors",
907
+ "model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00019.safetensors",
908
+ "model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00019.safetensors",
909
+ "model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00019.safetensors",
910
+ "model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00019.safetensors",
911
+ "model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00019.safetensors",
912
+ "model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00019.safetensors",
913
+ "model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00019.safetensors",
914
+ "model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00019.safetensors",
915
+ "model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00019.safetensors",
916
+ "model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00019.safetensors",
917
+ "model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00019.safetensors",
918
+ "model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00019.safetensors",
919
+ "model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00019.safetensors",
920
+ "model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00019.safetensors",
921
+ "model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00019.safetensors",
922
+ "model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00019.safetensors",
923
+ "model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00019.safetensors",
924
+ "model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00019.safetensors",
925
+ "model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00019.safetensors",
926
+ "model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00019.safetensors",
927
+ "model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00019.safetensors",
928
+ "model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00019.safetensors",
929
+ "model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00019.safetensors",
930
+ "model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00019.safetensors",
931
+ "model.layers.7.block_sparse_moe.gate.weight": "model-00005-of-00019.safetensors",
932
+ "model.layers.7.input_layernorm.weight": "model-00005-of-00019.safetensors",
933
+ "model.layers.7.post_attention_layernorm.weight": "model-00005-of-00019.safetensors",
934
+ "model.layers.7.self_attn.k_proj.weight": "model-00005-of-00019.safetensors",
935
+ "model.layers.7.self_attn.o_proj.weight": "model-00005-of-00019.safetensors",
936
+ "model.layers.7.self_attn.q_proj.weight": "model-00005-of-00019.safetensors",
937
+ "model.layers.7.self_attn.v_proj.weight": "model-00005-of-00019.safetensors",
938
+ "model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00019.safetensors",
939
+ "model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00019.safetensors",
940
+ "model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00019.safetensors",
941
+ "model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00019.safetensors",
942
+ "model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00019.safetensors",
943
+ "model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00019.safetensors",
944
+ "model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00019.safetensors",
945
+ "model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00019.safetensors",
946
+ "model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00019.safetensors",
947
+ "model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00019.safetensors",
948
+ "model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00019.safetensors",
949
+ "model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00019.safetensors",
950
+ "model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00019.safetensors",
951
+ "model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00019.safetensors",
952
+ "model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00019.safetensors",
953
+ "model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00019.safetensors",
954
+ "model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00019.safetensors",
955
+ "model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00019.safetensors",
956
+ "model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00019.safetensors",
957
+ "model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00019.safetensors",
958
+ "model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00019.safetensors",
959
+ "model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00019.safetensors",
960
+ "model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00019.safetensors",
961
+ "model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00019.safetensors",
962
+ "model.layers.8.block_sparse_moe.gate.weight": "model-00005-of-00019.safetensors",
963
+ "model.layers.8.input_layernorm.weight": "model-00006-of-00019.safetensors",
964
+ "model.layers.8.post_attention_layernorm.weight": "model-00006-of-00019.safetensors",
965
+ "model.layers.8.self_attn.k_proj.weight": "model-00005-of-00019.safetensors",
966
+ "model.layers.8.self_attn.o_proj.weight": "model-00005-of-00019.safetensors",
967
+ "model.layers.8.self_attn.q_proj.weight": "model-00005-of-00019.safetensors",
968
+ "model.layers.8.self_attn.v_proj.weight": "model-00005-of-00019.safetensors",
969
+ "model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00019.safetensors",
970
+ "model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00019.safetensors",
971
+ "model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00019.safetensors",
972
+ "model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00019.safetensors",
973
+ "model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00019.safetensors",
974
+ "model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00019.safetensors",
975
+ "model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00019.safetensors",
976
+ "model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00019.safetensors",
977
+ "model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00019.safetensors",
978
+ "model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00019.safetensors",
979
+ "model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00019.safetensors",
980
+ "model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00019.safetensors",
981
+ "model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00019.safetensors",
982
+ "model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00019.safetensors",
983
+ "model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00019.safetensors",
984
+ "model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00019.safetensors",
985
+ "model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00019.safetensors",
986
+ "model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00019.safetensors",
987
+ "model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00019.safetensors",
988
+ "model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00019.safetensors",
989
+ "model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00019.safetensors",
990
+ "model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00019.safetensors",
991
+ "model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00019.safetensors",
992
+ "model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00019.safetensors",
993
+ "model.layers.9.block_sparse_moe.gate.weight": "model-00006-of-00019.safetensors",
994
+ "model.layers.9.input_layernorm.weight": "model-00006-of-00019.safetensors",
995
+ "model.layers.9.post_attention_layernorm.weight": "model-00006-of-00019.safetensors",
996
+ "model.layers.9.self_attn.k_proj.weight": "model-00006-of-00019.safetensors",
997
+ "model.layers.9.self_attn.o_proj.weight": "model-00006-of-00019.safetensors",
998
+ "model.layers.9.self_attn.q_proj.weight": "model-00006-of-00019.safetensors",
999
+ "model.layers.9.self_attn.v_proj.weight": "model-00006-of-00019.safetensors",
1000
+ "model.norm.weight": "model-00019-of-00019.safetensors"
1001
+ }
1002
+ }
output-00001-of-00002.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3c10e16417e02d3d4c160b9b7186b714c0e48f26782b0e1bfe988b78e3d7c7bf
3
+ size 8581503776
output-00002-of-00002.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7e93249357c1eaaaccd3425591ef58963407ff551efe184391af39184378954
3
+ size 5744608776
special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "</s>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "0": {
6
+ "content": "<unk>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "1": {
14
+ "content": "<s>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "2": {
22
+ "content": "</s>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ }
29
+ },
30
+ "additional_special_tokens": [],
31
+ "bos_token": "<s>",
32
+ "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}",
33
+ "clean_up_tokenization_spaces": false,
34
+ "eos_token": "</s>",
35
+ "legacy": true,
36
+ "model_max_length": 2048,
37
+ "pad_token": "</s>",
38
+ "sp_model_kwargs": {},
39
+ "spaces_between_special_tokens": false,
40
+ "tokenizer_class": "LlamaTokenizer",
41
+ "unk_token": "<unk>",
42
+ "use_default_system_prompt": false
43
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.4461688995361328,
4
+ "train_runtime": 44067.2139,
5
+ "train_samples": 58917,
6
+ "train_samples_per_second": 1.337,
7
+ "train_steps_per_second": 0.021
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,1396 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 1.0,
5
+ "eval_steps": 200,
6
+ "global_step": 921,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0,
13
+ "learning_rate": 5.3763440860215056e-09,
14
+ "logits/chosen": -1.3876744508743286,
15
+ "logits/rejected": -1.355853796005249,
16
+ "logps/chosen": -672.8331298828125,
17
+ "logps/rejected": -549.942138671875,
18
+ "loss": 0.6931,
19
+ "rewards/accuracies": 0.0,
20
+ "rewards/chosen": 0.0,
21
+ "rewards/margins": 0.0,
22
+ "rewards/rejected": 0.0,
23
+ "step": 1
24
+ },
25
+ {
26
+ "epoch": 0.01,
27
+ "learning_rate": 5.3763440860215054e-08,
28
+ "logits/chosen": -1.1944549083709717,
29
+ "logits/rejected": -1.1097763776779175,
30
+ "logps/chosen": -368.380859375,
31
+ "logps/rejected": -353.98101806640625,
32
+ "loss": 0.709,
33
+ "rewards/accuracies": 0.4305555522441864,
34
+ "rewards/chosen": 0.015228834003210068,
35
+ "rewards/margins": 0.0040851375088095665,
36
+ "rewards/rejected": 0.011143695563077927,
37
+ "step": 10
38
+ },
39
+ {
40
+ "epoch": 0.02,
41
+ "learning_rate": 1.0752688172043011e-07,
42
+ "logits/chosen": -1.4370033740997314,
43
+ "logits/rejected": -1.4178457260131836,
44
+ "logps/chosen": -391.20050048828125,
45
+ "logps/rejected": -409.3836364746094,
46
+ "loss": 0.7198,
47
+ "rewards/accuracies": 0.512499988079071,
48
+ "rewards/chosen": -0.015771254897117615,
49
+ "rewards/margins": 0.035423360764980316,
50
+ "rewards/rejected": -0.05119461938738823,
51
+ "step": 20
52
+ },
53
+ {
54
+ "epoch": 0.03,
55
+ "learning_rate": 1.6129032258064515e-07,
56
+ "logits/chosen": -1.416802167892456,
57
+ "logits/rejected": -1.4156275987625122,
58
+ "logps/chosen": -411.7056579589844,
59
+ "logps/rejected": -425.6934509277344,
60
+ "loss": 0.6959,
61
+ "rewards/accuracies": 0.5625,
62
+ "rewards/chosen": -0.15962687134742737,
63
+ "rewards/margins": 0.045185185968875885,
64
+ "rewards/rejected": -0.20481204986572266,
65
+ "step": 30
66
+ },
67
+ {
68
+ "epoch": 0.04,
69
+ "learning_rate": 2.1505376344086022e-07,
70
+ "logits/chosen": -1.2398654222488403,
71
+ "logits/rejected": -1.2564724683761597,
72
+ "logps/chosen": -385.45782470703125,
73
+ "logps/rejected": -452.55535888671875,
74
+ "loss": 0.6365,
75
+ "rewards/accuracies": 0.7124999761581421,
76
+ "rewards/chosen": -0.3178340792655945,
77
+ "rewards/margins": 0.27168726921081543,
78
+ "rewards/rejected": -0.5895214080810547,
79
+ "step": 40
80
+ },
81
+ {
82
+ "epoch": 0.05,
83
+ "learning_rate": 2.6881720430107523e-07,
84
+ "logits/chosen": -1.2720705270767212,
85
+ "logits/rejected": -0.9971787333488464,
86
+ "logps/chosen": -339.6200866699219,
87
+ "logps/rejected": -361.6040954589844,
88
+ "loss": 0.5605,
89
+ "rewards/accuracies": 0.6625000238418579,
90
+ "rewards/chosen": -0.790237307548523,
91
+ "rewards/margins": 0.61395663022995,
92
+ "rewards/rejected": -1.4041939973831177,
93
+ "step": 50
94
+ },
95
+ {
96
+ "epoch": 0.07,
97
+ "learning_rate": 3.225806451612903e-07,
98
+ "logits/chosen": -1.350678563117981,
99
+ "logits/rejected": -1.3656706809997559,
100
+ "logps/chosen": -417.0431213378906,
101
+ "logps/rejected": -416.76165771484375,
102
+ "loss": 0.5485,
103
+ "rewards/accuracies": 0.75,
104
+ "rewards/chosen": -1.473722219467163,
105
+ "rewards/margins": 1.0274063348770142,
106
+ "rewards/rejected": -2.5011284351348877,
107
+ "step": 60
108
+ },
109
+ {
110
+ "epoch": 0.08,
111
+ "learning_rate": 3.7634408602150537e-07,
112
+ "logits/chosen": -1.2629830837249756,
113
+ "logits/rejected": -1.415208339691162,
114
+ "logps/chosen": -377.01275634765625,
115
+ "logps/rejected": -445.7503967285156,
116
+ "loss": 0.5203,
117
+ "rewards/accuracies": 0.675000011920929,
118
+ "rewards/chosen": -1.3981420993804932,
119
+ "rewards/margins": 0.6980546116828918,
120
+ "rewards/rejected": -2.0961966514587402,
121
+ "step": 70
122
+ },
123
+ {
124
+ "epoch": 0.09,
125
+ "learning_rate": 4.3010752688172043e-07,
126
+ "logits/chosen": -1.3798706531524658,
127
+ "logits/rejected": -1.3563942909240723,
128
+ "logps/chosen": -411.2007751464844,
129
+ "logps/rejected": -434.94476318359375,
130
+ "loss": 0.5067,
131
+ "rewards/accuracies": 0.762499988079071,
132
+ "rewards/chosen": -0.8918856382369995,
133
+ "rewards/margins": 1.0622823238372803,
134
+ "rewards/rejected": -1.9541680812835693,
135
+ "step": 80
136
+ },
137
+ {
138
+ "epoch": 0.1,
139
+ "learning_rate": 4.838709677419355e-07,
140
+ "logits/chosen": -1.6255207061767578,
141
+ "logits/rejected": -1.610365629196167,
142
+ "logps/chosen": -355.00128173828125,
143
+ "logps/rejected": -383.69866943359375,
144
+ "loss": 0.511,
145
+ "rewards/accuracies": 0.7250000238418579,
146
+ "rewards/chosen": -0.8219491839408875,
147
+ "rewards/margins": 0.6381314992904663,
148
+ "rewards/rejected": -1.460080623626709,
149
+ "step": 90
150
+ },
151
+ {
152
+ "epoch": 0.11,
153
+ "learning_rate": 4.957729468599034e-07,
154
+ "logits/chosen": -1.3692975044250488,
155
+ "logits/rejected": -1.3177874088287354,
156
+ "logps/chosen": -378.91265869140625,
157
+ "logps/rejected": -361.2559509277344,
158
+ "loss": 0.4586,
159
+ "rewards/accuracies": 0.7749999761581421,
160
+ "rewards/chosen": -0.2890338897705078,
161
+ "rewards/margins": 1.1477439403533936,
162
+ "rewards/rejected": -1.4367780685424805,
163
+ "step": 100
164
+ },
165
+ {
166
+ "epoch": 0.12,
167
+ "learning_rate": 4.897342995169082e-07,
168
+ "logits/chosen": -1.3526300191879272,
169
+ "logits/rejected": -1.2792448997497559,
170
+ "logps/chosen": -378.30126953125,
171
+ "logps/rejected": -415.57464599609375,
172
+ "loss": 0.4725,
173
+ "rewards/accuracies": 0.762499988079071,
174
+ "rewards/chosen": -0.5904244184494019,
175
+ "rewards/margins": 1.3282153606414795,
176
+ "rewards/rejected": -1.9186397790908813,
177
+ "step": 110
178
+ },
179
+ {
180
+ "epoch": 0.13,
181
+ "learning_rate": 4.83695652173913e-07,
182
+ "logits/chosen": -1.4887703657150269,
183
+ "logits/rejected": -1.3336336612701416,
184
+ "logps/chosen": -374.025390625,
185
+ "logps/rejected": -446.5819396972656,
186
+ "loss": 0.4726,
187
+ "rewards/accuracies": 0.8125,
188
+ "rewards/chosen": -0.5127348899841309,
189
+ "rewards/margins": 1.7226130962371826,
190
+ "rewards/rejected": -2.2353477478027344,
191
+ "step": 120
192
+ },
193
+ {
194
+ "epoch": 0.14,
195
+ "learning_rate": 4.776570048309178e-07,
196
+ "logits/chosen": -1.2795524597167969,
197
+ "logits/rejected": -1.3249471187591553,
198
+ "logps/chosen": -375.96710205078125,
199
+ "logps/rejected": -457.1031188964844,
200
+ "loss": 0.4781,
201
+ "rewards/accuracies": 0.7749999761581421,
202
+ "rewards/chosen": -0.473715215921402,
203
+ "rewards/margins": 1.3638942241668701,
204
+ "rewards/rejected": -1.8376095294952393,
205
+ "step": 130
206
+ },
207
+ {
208
+ "epoch": 0.15,
209
+ "learning_rate": 4.716183574879227e-07,
210
+ "logits/chosen": -0.8936523199081421,
211
+ "logits/rejected": -0.731735348701477,
212
+ "logps/chosen": -377.02410888671875,
213
+ "logps/rejected": -374.2434387207031,
214
+ "loss": 0.4771,
215
+ "rewards/accuracies": 0.7749999761581421,
216
+ "rewards/chosen": -0.4292902946472168,
217
+ "rewards/margins": 1.5224840641021729,
218
+ "rewards/rejected": -1.9517742395401,
219
+ "step": 140
220
+ },
221
+ {
222
+ "epoch": 0.16,
223
+ "learning_rate": 4.655797101449275e-07,
224
+ "logits/chosen": -1.3006255626678467,
225
+ "logits/rejected": -1.1428701877593994,
226
+ "logps/chosen": -422.60986328125,
227
+ "logps/rejected": -394.9052429199219,
228
+ "loss": 0.4141,
229
+ "rewards/accuracies": 0.7250000238418579,
230
+ "rewards/chosen": -0.40644779801368713,
231
+ "rewards/margins": 1.319599986076355,
232
+ "rewards/rejected": -1.7260475158691406,
233
+ "step": 150
234
+ },
235
+ {
236
+ "epoch": 0.17,
237
+ "learning_rate": 4.5954106280193235e-07,
238
+ "logits/chosen": -1.3789948225021362,
239
+ "logits/rejected": -1.2863258123397827,
240
+ "logps/chosen": -305.4602355957031,
241
+ "logps/rejected": -374.1790466308594,
242
+ "loss": 0.4965,
243
+ "rewards/accuracies": 0.7749999761581421,
244
+ "rewards/chosen": -0.9782305955886841,
245
+ "rewards/margins": 1.459176778793335,
246
+ "rewards/rejected": -2.4374070167541504,
247
+ "step": 160
248
+ },
249
+ {
250
+ "epoch": 0.18,
251
+ "learning_rate": 4.5350241545893717e-07,
252
+ "logits/chosen": -1.4751253128051758,
253
+ "logits/rejected": -1.1761484146118164,
254
+ "logps/chosen": -395.0107727050781,
255
+ "logps/rejected": -390.03076171875,
256
+ "loss": 0.441,
257
+ "rewards/accuracies": 0.800000011920929,
258
+ "rewards/chosen": -0.5482892394065857,
259
+ "rewards/margins": 1.585138201713562,
260
+ "rewards/rejected": -2.133427143096924,
261
+ "step": 170
262
+ },
263
+ {
264
+ "epoch": 0.2,
265
+ "learning_rate": 4.47463768115942e-07,
266
+ "logits/chosen": -1.6447718143463135,
267
+ "logits/rejected": -1.6261088848114014,
268
+ "logps/chosen": -430.76129150390625,
269
+ "logps/rejected": -425.74969482421875,
270
+ "loss": 0.4392,
271
+ "rewards/accuracies": 0.762499988079071,
272
+ "rewards/chosen": -0.5816106796264648,
273
+ "rewards/margins": 1.8813564777374268,
274
+ "rewards/rejected": -2.4629669189453125,
275
+ "step": 180
276
+ },
277
+ {
278
+ "epoch": 0.21,
279
+ "learning_rate": 4.414251207729469e-07,
280
+ "logits/chosen": -1.2417869567871094,
281
+ "logits/rejected": -1.3334053754806519,
282
+ "logps/chosen": -337.415771484375,
283
+ "logps/rejected": -356.4654235839844,
284
+ "loss": 0.4574,
285
+ "rewards/accuracies": 0.737500011920929,
286
+ "rewards/chosen": -0.755621075630188,
287
+ "rewards/margins": 1.5105997323989868,
288
+ "rewards/rejected": -2.266220808029175,
289
+ "step": 190
290
+ },
291
+ {
292
+ "epoch": 0.22,
293
+ "learning_rate": 4.3538647342995165e-07,
294
+ "logits/chosen": -1.5782802104949951,
295
+ "logits/rejected": -1.6571115255355835,
296
+ "logps/chosen": -431.74151611328125,
297
+ "logps/rejected": -431.4637145996094,
298
+ "loss": 0.4384,
299
+ "rewards/accuracies": 0.8125,
300
+ "rewards/chosen": -0.07804969698190689,
301
+ "rewards/margins": 1.6836214065551758,
302
+ "rewards/rejected": -1.7616710662841797,
303
+ "step": 200
304
+ },
305
+ {
306
+ "epoch": 0.22,
307
+ "eval_logits/chosen": -1.4511131048202515,
308
+ "eval_logits/rejected": -1.3157293796539307,
309
+ "eval_logps/chosen": -397.8616638183594,
310
+ "eval_logps/rejected": -405.7994079589844,
311
+ "eval_loss": 0.4556037187576294,
312
+ "eval_rewards/accuracies": 0.7936508059501648,
313
+ "eval_rewards/chosen": -0.3274631202220917,
314
+ "eval_rewards/margins": 1.6173655986785889,
315
+ "eval_rewards/rejected": -1.944828748703003,
316
+ "eval_runtime": 405.1145,
317
+ "eval_samples_per_second": 4.937,
318
+ "eval_steps_per_second": 0.156,
319
+ "step": 200
320
+ },
321
+ {
322
+ "epoch": 0.23,
323
+ "learning_rate": 4.2934782608695653e-07,
324
+ "logits/chosen": -0.9520143270492554,
325
+ "logits/rejected": -1.01302969455719,
326
+ "logps/chosen": -444.7759704589844,
327
+ "logps/rejected": -401.30389404296875,
328
+ "loss": 0.4141,
329
+ "rewards/accuracies": 0.800000011920929,
330
+ "rewards/chosen": -0.38320282101631165,
331
+ "rewards/margins": 1.236628770828247,
332
+ "rewards/rejected": -1.6198316812515259,
333
+ "step": 210
334
+ },
335
+ {
336
+ "epoch": 0.24,
337
+ "learning_rate": 4.2330917874396135e-07,
338
+ "logits/chosen": -1.601231336593628,
339
+ "logits/rejected": -1.4172089099884033,
340
+ "logps/chosen": -431.98272705078125,
341
+ "logps/rejected": -412.7859802246094,
342
+ "loss": 0.4817,
343
+ "rewards/accuracies": 0.7875000238418579,
344
+ "rewards/chosen": -0.31900420784950256,
345
+ "rewards/margins": 1.7263078689575195,
346
+ "rewards/rejected": -2.0453121662139893,
347
+ "step": 220
348
+ },
349
+ {
350
+ "epoch": 0.25,
351
+ "learning_rate": 4.172705314009662e-07,
352
+ "logits/chosen": -1.1527206897735596,
353
+ "logits/rejected": -0.8433942794799805,
354
+ "logps/chosen": -383.67559814453125,
355
+ "logps/rejected": -369.19342041015625,
356
+ "loss": 0.45,
357
+ "rewards/accuracies": 0.7749999761581421,
358
+ "rewards/chosen": -0.5148889422416687,
359
+ "rewards/margins": 1.6342861652374268,
360
+ "rewards/rejected": -2.1491751670837402,
361
+ "step": 230
362
+ },
363
+ {
364
+ "epoch": 0.26,
365
+ "learning_rate": 4.11231884057971e-07,
366
+ "logits/chosen": -1.3957723379135132,
367
+ "logits/rejected": -1.4669153690338135,
368
+ "logps/chosen": -366.89093017578125,
369
+ "logps/rejected": -381.7056579589844,
370
+ "loss": 0.4573,
371
+ "rewards/accuracies": 0.762499988079071,
372
+ "rewards/chosen": -0.2394193708896637,
373
+ "rewards/margins": 1.0727304220199585,
374
+ "rewards/rejected": -1.3121497631072998,
375
+ "step": 240
376
+ },
377
+ {
378
+ "epoch": 0.27,
379
+ "learning_rate": 4.0519323671497583e-07,
380
+ "logits/chosen": -1.1060564517974854,
381
+ "logits/rejected": -1.020811676979065,
382
+ "logps/chosen": -431.10791015625,
383
+ "logps/rejected": -398.845458984375,
384
+ "loss": 0.4008,
385
+ "rewards/accuracies": 0.8374999761581421,
386
+ "rewards/chosen": -0.08395320922136307,
387
+ "rewards/margins": 1.6107537746429443,
388
+ "rewards/rejected": -1.694706916809082,
389
+ "step": 250
390
+ },
391
+ {
392
+ "epoch": 0.28,
393
+ "learning_rate": 3.9915458937198065e-07,
394
+ "logits/chosen": -1.0142405033111572,
395
+ "logits/rejected": -1.0058037042617798,
396
+ "logps/chosen": -404.4456481933594,
397
+ "logps/rejected": -417.4208984375,
398
+ "loss": 0.4342,
399
+ "rewards/accuracies": 0.8125,
400
+ "rewards/chosen": 0.09296660125255585,
401
+ "rewards/margins": 1.8066604137420654,
402
+ "rewards/rejected": -1.713693618774414,
403
+ "step": 260
404
+ },
405
+ {
406
+ "epoch": 0.29,
407
+ "learning_rate": 3.9311594202898553e-07,
408
+ "logits/chosen": -0.816851794719696,
409
+ "logits/rejected": -0.7470394372940063,
410
+ "logps/chosen": -395.8139343261719,
411
+ "logps/rejected": -355.42071533203125,
412
+ "loss": 0.4899,
413
+ "rewards/accuracies": 0.7749999761581421,
414
+ "rewards/chosen": -0.38039931654930115,
415
+ "rewards/margins": 1.4586222171783447,
416
+ "rewards/rejected": -1.8390214443206787,
417
+ "step": 270
418
+ },
419
+ {
420
+ "epoch": 0.3,
421
+ "learning_rate": 3.870772946859903e-07,
422
+ "logits/chosen": -1.0235198736190796,
423
+ "logits/rejected": -1.009104609489441,
424
+ "logps/chosen": -419.8531188964844,
425
+ "logps/rejected": -417.8365783691406,
426
+ "loss": 0.4258,
427
+ "rewards/accuracies": 0.800000011920929,
428
+ "rewards/chosen": -0.24367530643939972,
429
+ "rewards/margins": 1.267002820968628,
430
+ "rewards/rejected": -1.5106781721115112,
431
+ "step": 280
432
+ },
433
+ {
434
+ "epoch": 0.31,
435
+ "learning_rate": 3.810386473429952e-07,
436
+ "logits/chosen": -1.1125423908233643,
437
+ "logits/rejected": -1.093120813369751,
438
+ "logps/chosen": -347.3122863769531,
439
+ "logps/rejected": -387.7191467285156,
440
+ "loss": 0.435,
441
+ "rewards/accuracies": 0.7875000238418579,
442
+ "rewards/chosen": 0.060557298362255096,
443
+ "rewards/margins": 1.8192393779754639,
444
+ "rewards/rejected": -1.7586820125579834,
445
+ "step": 290
446
+ },
447
+ {
448
+ "epoch": 0.33,
449
+ "learning_rate": 3.75e-07,
450
+ "logits/chosen": -1.1970382928848267,
451
+ "logits/rejected": -1.1414059400558472,
452
+ "logps/chosen": -380.08892822265625,
453
+ "logps/rejected": -393.11126708984375,
454
+ "loss": 0.4137,
455
+ "rewards/accuracies": 0.737500011920929,
456
+ "rewards/chosen": -0.4049530625343323,
457
+ "rewards/margins": 1.5264403820037842,
458
+ "rewards/rejected": -1.9313932657241821,
459
+ "step": 300
460
+ },
461
+ {
462
+ "epoch": 0.34,
463
+ "learning_rate": 3.689613526570048e-07,
464
+ "logits/chosen": -1.4467629194259644,
465
+ "logits/rejected": -1.4420421123504639,
466
+ "logps/chosen": -412.6943359375,
467
+ "logps/rejected": -411.64031982421875,
468
+ "loss": 0.4545,
469
+ "rewards/accuracies": 0.75,
470
+ "rewards/chosen": -0.44115525484085083,
471
+ "rewards/margins": 1.5014537572860718,
472
+ "rewards/rejected": -1.9426085948944092,
473
+ "step": 310
474
+ },
475
+ {
476
+ "epoch": 0.35,
477
+ "learning_rate": 3.6292270531400966e-07,
478
+ "logits/chosen": -1.576790690422058,
479
+ "logits/rejected": -1.605373740196228,
480
+ "logps/chosen": -442.7998046875,
481
+ "logps/rejected": -471.8130798339844,
482
+ "loss": 0.426,
483
+ "rewards/accuracies": 0.7749999761581421,
484
+ "rewards/chosen": -0.1621936857700348,
485
+ "rewards/margins": 1.6487839221954346,
486
+ "rewards/rejected": -1.810977578163147,
487
+ "step": 320
488
+ },
489
+ {
490
+ "epoch": 0.36,
491
+ "learning_rate": 3.5688405797101443e-07,
492
+ "logits/chosen": -1.3066380023956299,
493
+ "logits/rejected": -1.1275885105133057,
494
+ "logps/chosen": -369.5173034667969,
495
+ "logps/rejected": -393.6142883300781,
496
+ "loss": 0.4594,
497
+ "rewards/accuracies": 0.737500011920929,
498
+ "rewards/chosen": -0.3911195993423462,
499
+ "rewards/margins": 1.561603307723999,
500
+ "rewards/rejected": -1.9527229070663452,
501
+ "step": 330
502
+ },
503
+ {
504
+ "epoch": 0.37,
505
+ "learning_rate": 3.508454106280193e-07,
506
+ "logits/chosen": -1.460533857345581,
507
+ "logits/rejected": -1.3913938999176025,
508
+ "logps/chosen": -461.4322814941406,
509
+ "logps/rejected": -443.2294006347656,
510
+ "loss": 0.4343,
511
+ "rewards/accuracies": 0.800000011920929,
512
+ "rewards/chosen": 0.05905343219637871,
513
+ "rewards/margins": 1.777876615524292,
514
+ "rewards/rejected": -1.7188230752944946,
515
+ "step": 340
516
+ },
517
+ {
518
+ "epoch": 0.38,
519
+ "learning_rate": 3.4480676328502414e-07,
520
+ "logits/chosen": -1.0480554103851318,
521
+ "logits/rejected": -0.939221203327179,
522
+ "logps/chosen": -389.5008544921875,
523
+ "logps/rejected": -391.26678466796875,
524
+ "loss": 0.4464,
525
+ "rewards/accuracies": 0.7749999761581421,
526
+ "rewards/chosen": -0.5172668099403381,
527
+ "rewards/margins": 1.5226786136627197,
528
+ "rewards/rejected": -2.039945125579834,
529
+ "step": 350
530
+ },
531
+ {
532
+ "epoch": 0.39,
533
+ "learning_rate": 3.3876811594202896e-07,
534
+ "logits/chosen": -1.367060899734497,
535
+ "logits/rejected": -1.250497579574585,
536
+ "logps/chosen": -374.1922302246094,
537
+ "logps/rejected": -377.8554382324219,
538
+ "loss": 0.4265,
539
+ "rewards/accuracies": 0.7749999761581421,
540
+ "rewards/chosen": -0.34825998544692993,
541
+ "rewards/margins": 1.6436443328857422,
542
+ "rewards/rejected": -1.9919040203094482,
543
+ "step": 360
544
+ },
545
+ {
546
+ "epoch": 0.4,
547
+ "learning_rate": 3.327294685990338e-07,
548
+ "logits/chosen": -1.4619824886322021,
549
+ "logits/rejected": -1.224457025527954,
550
+ "logps/chosen": -394.55694580078125,
551
+ "logps/rejected": -406.80389404296875,
552
+ "loss": 0.4343,
553
+ "rewards/accuracies": 0.8125,
554
+ "rewards/chosen": 0.11512075364589691,
555
+ "rewards/margins": 1.9784886837005615,
556
+ "rewards/rejected": -1.863368034362793,
557
+ "step": 370
558
+ },
559
+ {
560
+ "epoch": 0.41,
561
+ "learning_rate": 3.266908212560386e-07,
562
+ "logits/chosen": -1.5472246408462524,
563
+ "logits/rejected": -1.3760802745819092,
564
+ "logps/chosen": -370.1455993652344,
565
+ "logps/rejected": -360.59735107421875,
566
+ "loss": 0.4365,
567
+ "rewards/accuracies": 0.737500011920929,
568
+ "rewards/chosen": -0.24435913562774658,
569
+ "rewards/margins": 1.2893067598342896,
570
+ "rewards/rejected": -1.5336658954620361,
571
+ "step": 380
572
+ },
573
+ {
574
+ "epoch": 0.42,
575
+ "learning_rate": 3.2065217391304344e-07,
576
+ "logits/chosen": -1.3668596744537354,
577
+ "logits/rejected": -1.2702538967132568,
578
+ "logps/chosen": -429.017822265625,
579
+ "logps/rejected": -415.56439208984375,
580
+ "loss": 0.4121,
581
+ "rewards/accuracies": 0.8125,
582
+ "rewards/chosen": -0.2810801863670349,
583
+ "rewards/margins": 1.7658510208129883,
584
+ "rewards/rejected": -2.046931028366089,
585
+ "step": 390
586
+ },
587
+ {
588
+ "epoch": 0.43,
589
+ "learning_rate": 3.146135265700483e-07,
590
+ "logits/chosen": -1.3646513223648071,
591
+ "logits/rejected": -1.072363257408142,
592
+ "logps/chosen": -390.0302429199219,
593
+ "logps/rejected": -395.0215148925781,
594
+ "loss": 0.4064,
595
+ "rewards/accuracies": 0.800000011920929,
596
+ "rewards/chosen": -0.38891416788101196,
597
+ "rewards/margins": 1.990821123123169,
598
+ "rewards/rejected": -2.379735231399536,
599
+ "step": 400
600
+ },
601
+ {
602
+ "epoch": 0.43,
603
+ "eval_logits/chosen": -0.6539027094841003,
604
+ "eval_logits/rejected": -0.7660450339317322,
605
+ "eval_logps/chosen": -396.7496032714844,
606
+ "eval_logps/rejected": -408.44091796875,
607
+ "eval_loss": 0.4285692274570465,
608
+ "eval_rewards/accuracies": 0.8253968358039856,
609
+ "eval_rewards/chosen": -0.21625187993049622,
610
+ "eval_rewards/margins": 1.9927276372909546,
611
+ "eval_rewards/rejected": -2.208979606628418,
612
+ "eval_runtime": 409.2634,
613
+ "eval_samples_per_second": 4.887,
614
+ "eval_steps_per_second": 0.154,
615
+ "step": 400
616
+ },
617
+ {
618
+ "epoch": 0.45,
619
+ "learning_rate": 3.085748792270531e-07,
620
+ "logits/chosen": -1.2127052545547485,
621
+ "logits/rejected": -1.2303555011749268,
622
+ "logps/chosen": -399.98858642578125,
623
+ "logps/rejected": -388.2366638183594,
624
+ "loss": 0.4587,
625
+ "rewards/accuracies": 0.8125,
626
+ "rewards/chosen": -0.6597892642021179,
627
+ "rewards/margins": 1.7150468826293945,
628
+ "rewards/rejected": -2.3748364448547363,
629
+ "step": 410
630
+ },
631
+ {
632
+ "epoch": 0.46,
633
+ "learning_rate": 3.0253623188405797e-07,
634
+ "logits/chosen": -1.6032695770263672,
635
+ "logits/rejected": -1.3187893629074097,
636
+ "logps/chosen": -347.73553466796875,
637
+ "logps/rejected": -366.84814453125,
638
+ "loss": 0.4217,
639
+ "rewards/accuracies": 0.7749999761581421,
640
+ "rewards/chosen": -0.20497450232505798,
641
+ "rewards/margins": 1.4589314460754395,
642
+ "rewards/rejected": -1.6639057397842407,
643
+ "step": 420
644
+ },
645
+ {
646
+ "epoch": 0.47,
647
+ "learning_rate": 2.964975845410628e-07,
648
+ "logits/chosen": -1.4543535709381104,
649
+ "logits/rejected": -1.2742624282836914,
650
+ "logps/chosen": -423.95111083984375,
651
+ "logps/rejected": -437.7804260253906,
652
+ "loss": 0.4479,
653
+ "rewards/accuracies": 0.8125,
654
+ "rewards/chosen": 0.25907284021377563,
655
+ "rewards/margins": 2.2142040729522705,
656
+ "rewards/rejected": -1.9551312923431396,
657
+ "step": 430
658
+ },
659
+ {
660
+ "epoch": 0.48,
661
+ "learning_rate": 2.904589371980676e-07,
662
+ "logits/chosen": -1.9760487079620361,
663
+ "logits/rejected": -1.7164779901504517,
664
+ "logps/chosen": -386.17193603515625,
665
+ "logps/rejected": -450.7064514160156,
666
+ "loss": 0.4361,
667
+ "rewards/accuracies": 0.75,
668
+ "rewards/chosen": -0.34018686413764954,
669
+ "rewards/margins": 1.6212705373764038,
670
+ "rewards/rejected": -1.9614572525024414,
671
+ "step": 440
672
+ },
673
+ {
674
+ "epoch": 0.49,
675
+ "learning_rate": 2.8442028985507245e-07,
676
+ "logits/chosen": -1.5040684938430786,
677
+ "logits/rejected": -1.4459855556488037,
678
+ "logps/chosen": -410.3436584472656,
679
+ "logps/rejected": -438.83837890625,
680
+ "loss": 0.4386,
681
+ "rewards/accuracies": 0.7875000238418579,
682
+ "rewards/chosen": -0.016602765768766403,
683
+ "rewards/margins": 1.8991400003433228,
684
+ "rewards/rejected": -1.9157428741455078,
685
+ "step": 450
686
+ },
687
+ {
688
+ "epoch": 0.5,
689
+ "learning_rate": 2.7838164251207727e-07,
690
+ "logits/chosen": -1.193297266960144,
691
+ "logits/rejected": -0.9360073804855347,
692
+ "logps/chosen": -415.88568115234375,
693
+ "logps/rejected": -389.2828674316406,
694
+ "loss": 0.4233,
695
+ "rewards/accuracies": 0.800000011920929,
696
+ "rewards/chosen": 0.08352740854024887,
697
+ "rewards/margins": 2.2130849361419678,
698
+ "rewards/rejected": -2.1295576095581055,
699
+ "step": 460
700
+ },
701
+ {
702
+ "epoch": 0.51,
703
+ "learning_rate": 2.723429951690821e-07,
704
+ "logits/chosen": -1.5409491062164307,
705
+ "logits/rejected": -1.7222496271133423,
706
+ "logps/chosen": -426.3505859375,
707
+ "logps/rejected": -415.15753173828125,
708
+ "loss": 0.3939,
709
+ "rewards/accuracies": 0.8500000238418579,
710
+ "rewards/chosen": -0.04843442887067795,
711
+ "rewards/margins": 2.11486554145813,
712
+ "rewards/rejected": -2.163300037384033,
713
+ "step": 470
714
+ },
715
+ {
716
+ "epoch": 0.52,
717
+ "learning_rate": 2.66304347826087e-07,
718
+ "logits/chosen": -1.1968003511428833,
719
+ "logits/rejected": -1.0116466283798218,
720
+ "logps/chosen": -381.65277099609375,
721
+ "logps/rejected": -407.2720642089844,
722
+ "loss": 0.4468,
723
+ "rewards/accuracies": 0.800000011920929,
724
+ "rewards/chosen": 0.10518859326839447,
725
+ "rewards/margins": 2.6760292053222656,
726
+ "rewards/rejected": -2.570840358734131,
727
+ "step": 480
728
+ },
729
+ {
730
+ "epoch": 0.53,
731
+ "learning_rate": 2.6026570048309175e-07,
732
+ "logits/chosen": -1.5621907711029053,
733
+ "logits/rejected": -1.4111429452896118,
734
+ "logps/chosen": -331.765380859375,
735
+ "logps/rejected": -345.84417724609375,
736
+ "loss": 0.4083,
737
+ "rewards/accuracies": 0.762499988079071,
738
+ "rewards/chosen": -0.2680264115333557,
739
+ "rewards/margins": 1.9610168933868408,
740
+ "rewards/rejected": -2.229043483734131,
741
+ "step": 490
742
+ },
743
+ {
744
+ "epoch": 0.54,
745
+ "learning_rate": 2.5422705314009663e-07,
746
+ "logits/chosen": -1.137390375137329,
747
+ "logits/rejected": -0.9998054504394531,
748
+ "logps/chosen": -393.7897644042969,
749
+ "logps/rejected": -426.21612548828125,
750
+ "loss": 0.4487,
751
+ "rewards/accuracies": 0.7124999761581421,
752
+ "rewards/chosen": -0.3346656262874603,
753
+ "rewards/margins": 1.5842936038970947,
754
+ "rewards/rejected": -1.9189590215682983,
755
+ "step": 500
756
+ },
757
+ {
758
+ "epoch": 0.55,
759
+ "learning_rate": 2.4818840579710145e-07,
760
+ "logits/chosen": -1.2915681600570679,
761
+ "logits/rejected": -1.1393409967422485,
762
+ "logps/chosen": -476.5746154785156,
763
+ "logps/rejected": -397.6057434082031,
764
+ "loss": 0.4763,
765
+ "rewards/accuracies": 0.762499988079071,
766
+ "rewards/chosen": -0.33603593707084656,
767
+ "rewards/margins": 1.7989603281021118,
768
+ "rewards/rejected": -2.134996175765991,
769
+ "step": 510
770
+ },
771
+ {
772
+ "epoch": 0.56,
773
+ "learning_rate": 2.421497584541063e-07,
774
+ "logits/chosen": -1.4665111303329468,
775
+ "logits/rejected": -1.4302623271942139,
776
+ "logps/chosen": -410.9242248535156,
777
+ "logps/rejected": -435.70721435546875,
778
+ "loss": 0.4378,
779
+ "rewards/accuracies": 0.75,
780
+ "rewards/chosen": -0.36040204763412476,
781
+ "rewards/margins": 1.7777122259140015,
782
+ "rewards/rejected": -2.1381144523620605,
783
+ "step": 520
784
+ },
785
+ {
786
+ "epoch": 0.58,
787
+ "learning_rate": 2.361111111111111e-07,
788
+ "logits/chosen": -1.3417316675186157,
789
+ "logits/rejected": -1.2472248077392578,
790
+ "logps/chosen": -396.3175354003906,
791
+ "logps/rejected": -406.9625244140625,
792
+ "loss": 0.3857,
793
+ "rewards/accuracies": 0.862500011920929,
794
+ "rewards/chosen": 0.186844140291214,
795
+ "rewards/margins": 2.318281650543213,
796
+ "rewards/rejected": -2.131437301635742,
797
+ "step": 530
798
+ },
799
+ {
800
+ "epoch": 0.59,
801
+ "learning_rate": 2.3007246376811593e-07,
802
+ "logits/chosen": -1.5599995851516724,
803
+ "logits/rejected": -1.4291542768478394,
804
+ "logps/chosen": -411.170654296875,
805
+ "logps/rejected": -423.13177490234375,
806
+ "loss": 0.4244,
807
+ "rewards/accuracies": 0.875,
808
+ "rewards/chosen": -0.1071757823228836,
809
+ "rewards/margins": 1.971361517906189,
810
+ "rewards/rejected": -2.0785374641418457,
811
+ "step": 540
812
+ },
813
+ {
814
+ "epoch": 0.6,
815
+ "learning_rate": 2.2403381642512075e-07,
816
+ "logits/chosen": -1.2990379333496094,
817
+ "logits/rejected": -1.1973732709884644,
818
+ "logps/chosen": -418.79437255859375,
819
+ "logps/rejected": -472.189453125,
820
+ "loss": 0.4643,
821
+ "rewards/accuracies": 0.8374999761581421,
822
+ "rewards/chosen": 0.147103950381279,
823
+ "rewards/margins": 2.323150157928467,
824
+ "rewards/rejected": -2.176046371459961,
825
+ "step": 550
826
+ },
827
+ {
828
+ "epoch": 0.61,
829
+ "learning_rate": 2.1799516908212558e-07,
830
+ "logits/chosen": -1.8142582178115845,
831
+ "logits/rejected": -1.725685715675354,
832
+ "logps/chosen": -374.6809387207031,
833
+ "logps/rejected": -410.0592346191406,
834
+ "loss": 0.4145,
835
+ "rewards/accuracies": 0.824999988079071,
836
+ "rewards/chosen": -0.23340897262096405,
837
+ "rewards/margins": 2.3226966857910156,
838
+ "rewards/rejected": -2.556105852127075,
839
+ "step": 560
840
+ },
841
+ {
842
+ "epoch": 0.62,
843
+ "learning_rate": 2.1195652173913043e-07,
844
+ "logits/chosen": -1.4505486488342285,
845
+ "logits/rejected": -1.3042548894882202,
846
+ "logps/chosen": -409.4254150390625,
847
+ "logps/rejected": -374.3670654296875,
848
+ "loss": 0.4178,
849
+ "rewards/accuracies": 0.8500000238418579,
850
+ "rewards/chosen": 0.1292637288570404,
851
+ "rewards/margins": 2.543600082397461,
852
+ "rewards/rejected": -2.414336681365967,
853
+ "step": 570
854
+ },
855
+ {
856
+ "epoch": 0.63,
857
+ "learning_rate": 2.0591787439613526e-07,
858
+ "logits/chosen": -0.5316571593284607,
859
+ "logits/rejected": -0.45167016983032227,
860
+ "logps/chosen": -386.24517822265625,
861
+ "logps/rejected": -434.59393310546875,
862
+ "loss": 0.3853,
863
+ "rewards/accuracies": 0.737500011920929,
864
+ "rewards/chosen": -0.4753696322441101,
865
+ "rewards/margins": 1.9324595928192139,
866
+ "rewards/rejected": -2.4078292846679688,
867
+ "step": 580
868
+ },
869
+ {
870
+ "epoch": 0.64,
871
+ "learning_rate": 1.9987922705314008e-07,
872
+ "logits/chosen": -1.3159304857254028,
873
+ "logits/rejected": -1.087425947189331,
874
+ "logps/chosen": -377.9801025390625,
875
+ "logps/rejected": -377.0760803222656,
876
+ "loss": 0.4461,
877
+ "rewards/accuracies": 0.824999988079071,
878
+ "rewards/chosen": -0.38399359583854675,
879
+ "rewards/margins": 2.1969008445739746,
880
+ "rewards/rejected": -2.580894708633423,
881
+ "step": 590
882
+ },
883
+ {
884
+ "epoch": 0.65,
885
+ "learning_rate": 1.938405797101449e-07,
886
+ "logits/chosen": -0.9003721475601196,
887
+ "logits/rejected": -0.6628047227859497,
888
+ "logps/chosen": -435.5120544433594,
889
+ "logps/rejected": -446.45550537109375,
890
+ "loss": 0.3952,
891
+ "rewards/accuracies": 0.875,
892
+ "rewards/chosen": 0.012645396403968334,
893
+ "rewards/margins": 1.9577815532684326,
894
+ "rewards/rejected": -1.9451364278793335,
895
+ "step": 600
896
+ },
897
+ {
898
+ "epoch": 0.65,
899
+ "eval_logits/chosen": -0.7205740809440613,
900
+ "eval_logits/rejected": -0.6783490777015686,
901
+ "eval_logps/chosen": -395.8981628417969,
902
+ "eval_logps/rejected": -407.95367431640625,
903
+ "eval_loss": 0.42750075459480286,
904
+ "eval_rewards/accuracies": 0.8015872836112976,
905
+ "eval_rewards/chosen": -0.13110622763633728,
906
+ "eval_rewards/margins": 2.0291495323181152,
907
+ "eval_rewards/rejected": -2.1602559089660645,
908
+ "eval_runtime": 404.1135,
909
+ "eval_samples_per_second": 4.949,
910
+ "eval_steps_per_second": 0.156,
911
+ "step": 600
912
+ },
913
+ {
914
+ "epoch": 0.66,
915
+ "learning_rate": 1.8780193236714976e-07,
916
+ "logits/chosen": -0.49158763885498047,
917
+ "logits/rejected": -0.30927062034606934,
918
+ "logps/chosen": -421.99005126953125,
919
+ "logps/rejected": -378.0501708984375,
920
+ "loss": 0.4263,
921
+ "rewards/accuracies": 0.7749999761581421,
922
+ "rewards/chosen": -0.14008215069770813,
923
+ "rewards/margins": 1.6353435516357422,
924
+ "rewards/rejected": -1.775425672531128,
925
+ "step": 610
926
+ },
927
+ {
928
+ "epoch": 0.67,
929
+ "learning_rate": 1.8176328502415459e-07,
930
+ "logits/chosen": -1.3565137386322021,
931
+ "logits/rejected": -1.1652584075927734,
932
+ "logps/chosen": -450.35540771484375,
933
+ "logps/rejected": -436.74761962890625,
934
+ "loss": 0.4252,
935
+ "rewards/accuracies": 0.7749999761581421,
936
+ "rewards/chosen": -0.03007013536989689,
937
+ "rewards/margins": 1.9853531122207642,
938
+ "rewards/rejected": -2.01542329788208,
939
+ "step": 620
940
+ },
941
+ {
942
+ "epoch": 0.68,
943
+ "learning_rate": 1.757246376811594e-07,
944
+ "logits/chosen": -0.015371406450867653,
945
+ "logits/rejected": -0.01580933667719364,
946
+ "logps/chosen": -382.2900390625,
947
+ "logps/rejected": -360.01446533203125,
948
+ "loss": 0.3848,
949
+ "rewards/accuracies": 0.949999988079071,
950
+ "rewards/chosen": 0.17353440821170807,
951
+ "rewards/margins": 2.286402940750122,
952
+ "rewards/rejected": -2.1128687858581543,
953
+ "step": 630
954
+ },
955
+ {
956
+ "epoch": 0.69,
957
+ "learning_rate": 1.6968599033816424e-07,
958
+ "logits/chosen": 0.3802258372306824,
959
+ "logits/rejected": 0.4669532775878906,
960
+ "logps/chosen": -355.05322265625,
961
+ "logps/rejected": -390.06976318359375,
962
+ "loss": 0.4137,
963
+ "rewards/accuracies": 0.737500011920929,
964
+ "rewards/chosen": -0.6333318948745728,
965
+ "rewards/margins": 1.6801398992538452,
966
+ "rewards/rejected": -2.313471794128418,
967
+ "step": 640
968
+ },
969
+ {
970
+ "epoch": 0.71,
971
+ "learning_rate": 1.636473429951691e-07,
972
+ "logits/chosen": 0.4362594485282898,
973
+ "logits/rejected": 0.5603054761886597,
974
+ "logps/chosen": -411.2711486816406,
975
+ "logps/rejected": -398.11346435546875,
976
+ "loss": 0.4552,
977
+ "rewards/accuracies": 0.800000011920929,
978
+ "rewards/chosen": -0.7625397443771362,
979
+ "rewards/margins": 1.8215090036392212,
980
+ "rewards/rejected": -2.5840485095977783,
981
+ "step": 650
982
+ },
983
+ {
984
+ "epoch": 0.72,
985
+ "learning_rate": 1.5760869565217392e-07,
986
+ "logits/chosen": 0.4388657212257385,
987
+ "logits/rejected": 0.8059428334236145,
988
+ "logps/chosen": -406.7152404785156,
989
+ "logps/rejected": -383.4807434082031,
990
+ "loss": 0.4061,
991
+ "rewards/accuracies": 0.7749999761581421,
992
+ "rewards/chosen": -0.5559307336807251,
993
+ "rewards/margins": 1.9154608249664307,
994
+ "rewards/rejected": -2.4713916778564453,
995
+ "step": 660
996
+ },
997
+ {
998
+ "epoch": 0.73,
999
+ "learning_rate": 1.5157004830917874e-07,
1000
+ "logits/chosen": 0.707136332988739,
1001
+ "logits/rejected": 0.8469961881637573,
1002
+ "logps/chosen": -373.77032470703125,
1003
+ "logps/rejected": -383.32525634765625,
1004
+ "loss": 0.4709,
1005
+ "rewards/accuracies": 0.762499988079071,
1006
+ "rewards/chosen": -0.159020334482193,
1007
+ "rewards/margins": 1.6874653100967407,
1008
+ "rewards/rejected": -1.8464854955673218,
1009
+ "step": 670
1010
+ },
1011
+ {
1012
+ "epoch": 0.74,
1013
+ "learning_rate": 1.4553140096618357e-07,
1014
+ "logits/chosen": -0.08916174620389938,
1015
+ "logits/rejected": -0.07962872833013535,
1016
+ "logps/chosen": -403.15765380859375,
1017
+ "logps/rejected": -382.7056884765625,
1018
+ "loss": 0.4123,
1019
+ "rewards/accuracies": 0.7875000238418579,
1020
+ "rewards/chosen": 0.1752249300479889,
1021
+ "rewards/margins": 2.1825637817382812,
1022
+ "rewards/rejected": -2.007338762283325,
1023
+ "step": 680
1024
+ },
1025
+ {
1026
+ "epoch": 0.75,
1027
+ "learning_rate": 1.3949275362318842e-07,
1028
+ "logits/chosen": 0.2861272692680359,
1029
+ "logits/rejected": 0.5865569114685059,
1030
+ "logps/chosen": -438.97808837890625,
1031
+ "logps/rejected": -424.977294921875,
1032
+ "loss": 0.3888,
1033
+ "rewards/accuracies": 0.737500011920929,
1034
+ "rewards/chosen": -0.10896565020084381,
1035
+ "rewards/margins": 1.5186336040496826,
1036
+ "rewards/rejected": -1.6275993585586548,
1037
+ "step": 690
1038
+ },
1039
+ {
1040
+ "epoch": 0.76,
1041
+ "learning_rate": 1.3345410628019324e-07,
1042
+ "logits/chosen": 0.7959792017936707,
1043
+ "logits/rejected": 1.0797499418258667,
1044
+ "logps/chosen": -399.54302978515625,
1045
+ "logps/rejected": -411.88824462890625,
1046
+ "loss": 0.4496,
1047
+ "rewards/accuracies": 0.737500011920929,
1048
+ "rewards/chosen": -0.02021363377571106,
1049
+ "rewards/margins": 2.1036648750305176,
1050
+ "rewards/rejected": -2.1238787174224854,
1051
+ "step": 700
1052
+ },
1053
+ {
1054
+ "epoch": 0.77,
1055
+ "learning_rate": 1.2741545893719807e-07,
1056
+ "logits/chosen": 0.061588358134031296,
1057
+ "logits/rejected": 0.5792427659034729,
1058
+ "logps/chosen": -389.0401611328125,
1059
+ "logps/rejected": -442.1222229003906,
1060
+ "loss": 0.461,
1061
+ "rewards/accuracies": 0.7875000238418579,
1062
+ "rewards/chosen": -0.3942088484764099,
1063
+ "rewards/margins": 1.9804880619049072,
1064
+ "rewards/rejected": -2.3746962547302246,
1065
+ "step": 710
1066
+ },
1067
+ {
1068
+ "epoch": 0.78,
1069
+ "learning_rate": 1.213768115942029e-07,
1070
+ "logits/chosen": -0.3789304494857788,
1071
+ "logits/rejected": -0.5976569056510925,
1072
+ "logps/chosen": -383.9620056152344,
1073
+ "logps/rejected": -411.39923095703125,
1074
+ "loss": 0.3725,
1075
+ "rewards/accuracies": 0.7875000238418579,
1076
+ "rewards/chosen": -0.1539289653301239,
1077
+ "rewards/margins": 2.0121874809265137,
1078
+ "rewards/rejected": -2.166116237640381,
1079
+ "step": 720
1080
+ },
1081
+ {
1082
+ "epoch": 0.79,
1083
+ "learning_rate": 1.1533816425120772e-07,
1084
+ "logits/chosen": -0.32424452900886536,
1085
+ "logits/rejected": 0.054877616465091705,
1086
+ "logps/chosen": -399.9856262207031,
1087
+ "logps/rejected": -433.40008544921875,
1088
+ "loss": 0.4335,
1089
+ "rewards/accuracies": 0.7749999761581421,
1090
+ "rewards/chosen": -0.20857331156730652,
1091
+ "rewards/margins": 1.7776143550872803,
1092
+ "rewards/rejected": -1.9861876964569092,
1093
+ "step": 730
1094
+ },
1095
+ {
1096
+ "epoch": 0.8,
1097
+ "learning_rate": 1.0929951690821256e-07,
1098
+ "logits/chosen": 0.7587161064147949,
1099
+ "logits/rejected": 0.4672706127166748,
1100
+ "logps/chosen": -394.1853942871094,
1101
+ "logps/rejected": -397.8048095703125,
1102
+ "loss": 0.4497,
1103
+ "rewards/accuracies": 0.8500000238418579,
1104
+ "rewards/chosen": -0.04534666985273361,
1105
+ "rewards/margins": 2.2218165397644043,
1106
+ "rewards/rejected": -2.267162799835205,
1107
+ "step": 740
1108
+ },
1109
+ {
1110
+ "epoch": 0.81,
1111
+ "learning_rate": 1.0326086956521738e-07,
1112
+ "logits/chosen": -0.007064342498779297,
1113
+ "logits/rejected": -0.10647717863321304,
1114
+ "logps/chosen": -451.4822692871094,
1115
+ "logps/rejected": -446.7669372558594,
1116
+ "loss": 0.4347,
1117
+ "rewards/accuracies": 0.8125,
1118
+ "rewards/chosen": -0.011669009923934937,
1119
+ "rewards/margins": 1.8043409585952759,
1120
+ "rewards/rejected": -1.8160098791122437,
1121
+ "step": 750
1122
+ },
1123
+ {
1124
+ "epoch": 0.83,
1125
+ "learning_rate": 9.722222222222222e-08,
1126
+ "logits/chosen": -0.4753515124320984,
1127
+ "logits/rejected": -0.590586245059967,
1128
+ "logps/chosen": -471.35137939453125,
1129
+ "logps/rejected": -436.88531494140625,
1130
+ "loss": 0.386,
1131
+ "rewards/accuracies": 0.8500000238418579,
1132
+ "rewards/chosen": 0.01547972857952118,
1133
+ "rewards/margins": 2.1215872764587402,
1134
+ "rewards/rejected": -2.106107473373413,
1135
+ "step": 760
1136
+ },
1137
+ {
1138
+ "epoch": 0.84,
1139
+ "learning_rate": 9.118357487922705e-08,
1140
+ "logits/chosen": -0.4006249010562897,
1141
+ "logits/rejected": -0.5747838616371155,
1142
+ "logps/chosen": -375.1439514160156,
1143
+ "logps/rejected": -383.0394287109375,
1144
+ "loss": 0.3963,
1145
+ "rewards/accuracies": 0.8125,
1146
+ "rewards/chosen": -0.3230701684951782,
1147
+ "rewards/margins": 2.1252005100250244,
1148
+ "rewards/rejected": -2.448270797729492,
1149
+ "step": 770
1150
+ },
1151
+ {
1152
+ "epoch": 0.85,
1153
+ "learning_rate": 8.514492753623189e-08,
1154
+ "logits/chosen": 0.18706265091896057,
1155
+ "logits/rejected": 0.14009952545166016,
1156
+ "logps/chosen": -381.85443115234375,
1157
+ "logps/rejected": -417.75146484375,
1158
+ "loss": 0.3904,
1159
+ "rewards/accuracies": 0.7875000238418579,
1160
+ "rewards/chosen": -0.3124622702598572,
1161
+ "rewards/margins": 1.5956885814666748,
1162
+ "rewards/rejected": -1.9081509113311768,
1163
+ "step": 780
1164
+ },
1165
+ {
1166
+ "epoch": 0.86,
1167
+ "learning_rate": 7.910628019323671e-08,
1168
+ "logits/chosen": 0.359174907207489,
1169
+ "logits/rejected": 0.3385276198387146,
1170
+ "logps/chosen": -429.9090270996094,
1171
+ "logps/rejected": -362.47857666015625,
1172
+ "loss": 0.4715,
1173
+ "rewards/accuracies": 0.762499988079071,
1174
+ "rewards/chosen": -0.23555055260658264,
1175
+ "rewards/margins": 1.2514636516571045,
1176
+ "rewards/rejected": -1.4870140552520752,
1177
+ "step": 790
1178
+ },
1179
+ {
1180
+ "epoch": 0.87,
1181
+ "learning_rate": 7.306763285024155e-08,
1182
+ "logits/chosen": -0.1668163537979126,
1183
+ "logits/rejected": 0.2065034657716751,
1184
+ "logps/chosen": -394.82257080078125,
1185
+ "logps/rejected": -394.52142333984375,
1186
+ "loss": 0.3909,
1187
+ "rewards/accuracies": 0.762499988079071,
1188
+ "rewards/chosen": -0.4898454248905182,
1189
+ "rewards/margins": 1.7736278772354126,
1190
+ "rewards/rejected": -2.2634730339050293,
1191
+ "step": 800
1192
+ },
1193
+ {
1194
+ "epoch": 0.87,
1195
+ "eval_logits/chosen": -0.773809552192688,
1196
+ "eval_logits/rejected": -0.84580397605896,
1197
+ "eval_logps/chosen": -396.8601989746094,
1198
+ "eval_logps/rejected": -409.49676513671875,
1199
+ "eval_loss": 0.41672879457473755,
1200
+ "eval_rewards/accuracies": 0.8134920597076416,
1201
+ "eval_rewards/chosen": -0.22731684148311615,
1202
+ "eval_rewards/margins": 2.0872485637664795,
1203
+ "eval_rewards/rejected": -2.3145651817321777,
1204
+ "eval_runtime": 391.0713,
1205
+ "eval_samples_per_second": 5.114,
1206
+ "eval_steps_per_second": 0.161,
1207
+ "step": 800
1208
+ },
1209
+ {
1210
+ "epoch": 0.88,
1211
+ "learning_rate": 6.702898550724638e-08,
1212
+ "logits/chosen": -0.8055841326713562,
1213
+ "logits/rejected": -0.8581466674804688,
1214
+ "logps/chosen": -434.89300537109375,
1215
+ "logps/rejected": -407.73486328125,
1216
+ "loss": 0.4191,
1217
+ "rewards/accuracies": 0.824999988079071,
1218
+ "rewards/chosen": -0.5859915018081665,
1219
+ "rewards/margins": 1.8876674175262451,
1220
+ "rewards/rejected": -2.473658800125122,
1221
+ "step": 810
1222
+ },
1223
+ {
1224
+ "epoch": 0.89,
1225
+ "learning_rate": 6.09903381642512e-08,
1226
+ "logits/chosen": -0.6149953603744507,
1227
+ "logits/rejected": -0.6610177755355835,
1228
+ "logps/chosen": -403.4602355957031,
1229
+ "logps/rejected": -412.859619140625,
1230
+ "loss": 0.3632,
1231
+ "rewards/accuracies": 0.8500000238418579,
1232
+ "rewards/chosen": 0.0069200992584228516,
1233
+ "rewards/margins": 2.4245333671569824,
1234
+ "rewards/rejected": -2.4176132678985596,
1235
+ "step": 820
1236
+ },
1237
+ {
1238
+ "epoch": 0.9,
1239
+ "learning_rate": 5.4951690821256036e-08,
1240
+ "logits/chosen": -0.7099069356918335,
1241
+ "logits/rejected": -0.5322675704956055,
1242
+ "logps/chosen": -477.89080810546875,
1243
+ "logps/rejected": -509.3751525878906,
1244
+ "loss": 0.4242,
1245
+ "rewards/accuracies": 0.862500011920929,
1246
+ "rewards/chosen": 0.04087377339601517,
1247
+ "rewards/margins": 2.5105957984924316,
1248
+ "rewards/rejected": -2.469722270965576,
1249
+ "step": 830
1250
+ },
1251
+ {
1252
+ "epoch": 0.91,
1253
+ "learning_rate": 4.891304347826087e-08,
1254
+ "logits/chosen": -0.8323551416397095,
1255
+ "logits/rejected": -0.5835217833518982,
1256
+ "logps/chosen": -344.49261474609375,
1257
+ "logps/rejected": -416.62738037109375,
1258
+ "loss": 0.4114,
1259
+ "rewards/accuracies": 0.7250000238418579,
1260
+ "rewards/chosen": -0.43376049399375916,
1261
+ "rewards/margins": 1.836294412612915,
1262
+ "rewards/rejected": -2.270055055618286,
1263
+ "step": 840
1264
+ },
1265
+ {
1266
+ "epoch": 0.92,
1267
+ "learning_rate": 4.28743961352657e-08,
1268
+ "logits/chosen": -0.8896607160568237,
1269
+ "logits/rejected": -0.9991308450698853,
1270
+ "logps/chosen": -429.5093688964844,
1271
+ "logps/rejected": -462.163330078125,
1272
+ "loss": 0.4213,
1273
+ "rewards/accuracies": 0.7749999761581421,
1274
+ "rewards/chosen": 0.14300577342510223,
1275
+ "rewards/margins": 2.3328614234924316,
1276
+ "rewards/rejected": -2.1898555755615234,
1277
+ "step": 850
1278
+ },
1279
+ {
1280
+ "epoch": 0.93,
1281
+ "learning_rate": 3.6835748792270526e-08,
1282
+ "logits/chosen": -1.5217692852020264,
1283
+ "logits/rejected": -1.4032458066940308,
1284
+ "logps/chosen": -445.5403747558594,
1285
+ "logps/rejected": -484.6028747558594,
1286
+ "loss": 0.4084,
1287
+ "rewards/accuracies": 0.8125,
1288
+ "rewards/chosen": -0.3494121730327606,
1289
+ "rewards/margins": 1.832707166671753,
1290
+ "rewards/rejected": -2.182119369506836,
1291
+ "step": 860
1292
+ },
1293
+ {
1294
+ "epoch": 0.94,
1295
+ "learning_rate": 3.0797101449275364e-08,
1296
+ "logits/chosen": -1.3731000423431396,
1297
+ "logits/rejected": -1.262548804283142,
1298
+ "logps/chosen": -421.68524169921875,
1299
+ "logps/rejected": -415.681884765625,
1300
+ "loss": 0.4108,
1301
+ "rewards/accuracies": 0.8125,
1302
+ "rewards/chosen": 0.045568324625492096,
1303
+ "rewards/margins": 2.3908283710479736,
1304
+ "rewards/rejected": -2.34525990486145,
1305
+ "step": 870
1306
+ },
1307
+ {
1308
+ "epoch": 0.96,
1309
+ "learning_rate": 2.475845410628019e-08,
1310
+ "logits/chosen": -1.434045433998108,
1311
+ "logits/rejected": -1.2689034938812256,
1312
+ "logps/chosen": -386.4149169921875,
1313
+ "logps/rejected": -396.9994812011719,
1314
+ "loss": 0.4121,
1315
+ "rewards/accuracies": 0.824999988079071,
1316
+ "rewards/chosen": -0.10745694488286972,
1317
+ "rewards/margins": 2.171525478363037,
1318
+ "rewards/rejected": -2.278982400894165,
1319
+ "step": 880
1320
+ },
1321
+ {
1322
+ "epoch": 0.97,
1323
+ "learning_rate": 1.8719806763285022e-08,
1324
+ "logits/chosen": -1.6423695087432861,
1325
+ "logits/rejected": -1.5149376392364502,
1326
+ "logps/chosen": -413.06158447265625,
1327
+ "logps/rejected": -437.2244567871094,
1328
+ "loss": 0.3935,
1329
+ "rewards/accuracies": 0.8125,
1330
+ "rewards/chosen": -0.15360334515571594,
1331
+ "rewards/margins": 2.0728042125701904,
1332
+ "rewards/rejected": -2.226407527923584,
1333
+ "step": 890
1334
+ },
1335
+ {
1336
+ "epoch": 0.98,
1337
+ "learning_rate": 1.2681159420289856e-08,
1338
+ "logits/chosen": -0.9922491312026978,
1339
+ "logits/rejected": -1.0585081577301025,
1340
+ "logps/chosen": -401.11627197265625,
1341
+ "logps/rejected": -402.2158508300781,
1342
+ "loss": 0.3954,
1343
+ "rewards/accuracies": 0.800000011920929,
1344
+ "rewards/chosen": -0.1775648146867752,
1345
+ "rewards/margins": 2.2243430614471436,
1346
+ "rewards/rejected": -2.4019076824188232,
1347
+ "step": 900
1348
+ },
1349
+ {
1350
+ "epoch": 0.99,
1351
+ "learning_rate": 6.642512077294686e-09,
1352
+ "logits/chosen": 0.17671330273151398,
1353
+ "logits/rejected": 0.08089754730463028,
1354
+ "logps/chosen": -331.13409423828125,
1355
+ "logps/rejected": -373.9659729003906,
1356
+ "loss": 0.3714,
1357
+ "rewards/accuracies": 0.7749999761581421,
1358
+ "rewards/chosen": -0.24092864990234375,
1359
+ "rewards/margins": 1.8302351236343384,
1360
+ "rewards/rejected": -2.0711638927459717,
1361
+ "step": 910
1362
+ },
1363
+ {
1364
+ "epoch": 1.0,
1365
+ "learning_rate": 6.038647342995168e-10,
1366
+ "logits/chosen": -0.6910379528999329,
1367
+ "logits/rejected": -0.7387306094169617,
1368
+ "logps/chosen": -440.9806213378906,
1369
+ "logps/rejected": -403.29083251953125,
1370
+ "loss": 0.3947,
1371
+ "rewards/accuracies": 0.8500000238418579,
1372
+ "rewards/chosen": -0.311662882566452,
1373
+ "rewards/margins": 2.3290438652038574,
1374
+ "rewards/rejected": -2.640706777572632,
1375
+ "step": 920
1376
+ },
1377
+ {
1378
+ "epoch": 1.0,
1379
+ "step": 921,
1380
+ "total_flos": 0.0,
1381
+ "train_loss": 0.4461688995361328,
1382
+ "train_runtime": 44067.2139,
1383
+ "train_samples_per_second": 1.337,
1384
+ "train_steps_per_second": 0.021
1385
+ }
1386
+ ],
1387
+ "logging_steps": 10,
1388
+ "max_steps": 921,
1389
+ "num_input_tokens_seen": 0,
1390
+ "num_train_epochs": 1,
1391
+ "save_steps": 500,
1392
+ "total_flos": 0.0,
1393
+ "train_batch_size": 8,
1394
+ "trial_name": null,
1395
+ "trial_params": null
1396
+ }