ChloeAuYeung commited on
Commit
a6e5daf
1 Parent(s): a88b991

upload files

Browse files
MODEL_LICENSE.pdf ADDED
Binary file (315 kB). View file
 
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+
4
+ inference: false
5
+
6
+ ---
7
+
8
+ # XVERSE-13B
9
+
10
+ ## 模型介绍
11
+
12
+ **XVERSE-13B** 是由深圳元象科技自主研发的支持多语言的大语言模型(Large Language Model),主要特点如下:
13
+
14
+ - **模型结构**:XVERSE-13B 使用主流 Decoder-only 的标准 Transformer 网络结构,支持 8K 的上下文长度(Context Length),为同尺寸模型中最长,能满足更长的多轮对话、知识问答与摘要等需求,模型应用场景更广泛。
15
+ - **训练数据**:构建了 1.4 万亿 token 的高质量、多样化的数据对模型进行充分训练,包含中、英、俄、西等 40 多种语言,通过精细化设置不同类型数据的采样比例,使得中英两种语言表现优异,也能兼顾其他语言效果。
16
+ - **分词**:基于 BPE(Byte-Pair Encoding)算法,使用上百 GB 语料训练了一个词表大小为 100,278 的分词器,能够同时支持多语言,而无需额外扩展词表。
17
+ - **训练框架**:经过对算子、通信、并行策略及调度等的优化,千卡集群上的峰值算力利用率达到 58.5%,达到业界前列。
18
+
19
+ ## Model Introduction
20
+
21
+ **XVERSE-13B** is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. Its key features are as follows:
22
+
23
+ - **Model Structure**: XVERSE-13B uses the mainstream Decoder-only Transformer network structure, supports 8k context length, the longest one among models of the same size, which can meet the need of longer multi-round dialogues, knowledge question-answering, and summarization. This makes the model more versatile in application scenarios.
24
+ - **Training Data**: The model has been thoroughly trained on a diversified and high-quality dataset consisting of 1.4 trillion of tokens, including more than 40 languages such as Chinese, English, Russian, and Spanish. The sampling ratio of different types of data is finely set, which makes the performance of Chinese and English excellent, and also takes into account the effect of other languages.
25
+ - **Tokenization**: Based on the BPE (Byte-Pair Encoding) algorithm, a tokenizer with a vocabulary size of 100,278 has been trained using hundreds of gigabytes of language data. This tokenizer is capable of supporting multilingual without the need for additional vocabulary expansion.
26
+ - **Training Framework**: Through optimizations in operators, communication, parallel strategies, and scheduling, the peak computing power utilization rate on a thousand-card cluster reaches 58.5%, which is one of the highest in the industry.
27
+
28
+ ## 评测结果
29
+
30
+ 为验证模型的各项能力,我们选取了多个学科综合能力评测集,包括 [MMLU](https://arxiv.org/abs/2009.03300)(英文)、 [C-Eval](https://cevalbenchmark.com/)(中文)、[AGIEval](https://arxiv.org/abs/2304.06364)(中英) 、[GAOKAO-Bench](https://github.com/OpenLMLab/GAOKAO-Bench)(中英)、[GAOKAO-English](https://github.com/ExpressAI/AI-Gaokao)(英文),评测结果如下:
31
+
32
+ | 模型\数据集 | MMLU | C-Eval | AGIEval<sup>1</sup> | GAOKAO-Bench<sup>1</sup> | GAOKAO-English<sup>1</sup> |
33
+ | :------------------------: | :--------------: | :--------------: | :-----------------: | :----------------------: | :------------------------: |
34
+ | Baichuan-13B | 51.6<sup>2</sup> | 53.6<sup>3</sup> | 40.5 | 45.9 | 56.9 |
35
+ | Llama-1-13B | 46.9<sup>4</sup> | 28.8 | 27.3 | 26.4 | 38.1 |
36
+ | Llama-2-13B | 54.8<sup>4</sup> | 35.6 | 33.4 | 35.4 | 60.6 |
37
+ | moss-moon-003-base (16B) | 24.7 | 33.1<sup>3</sup> | 26.8 | 28.5 | 34.7 |
38
+ | OpenLLaMA-13B | 42.4 | 24.7 | 24.0 | 25.6 | 33.3 |
39
+ | OPT-13B | 25.2 | 25.0 | 24.2 | 24.4 | 31.1 |
40
+ | Pythia-12B | 25.1 | 26.2 | 25.3 | 25.3 | 26.8 |
41
+ | Ziya-LLaMA-13B-Pretrain-v1 | 43.9 | 30.2 | 27.2 | 26.4 | 37.6 |
42
+ | **XVERSE-13B** | **55.1** | **54.7** | **41.4** | **53.9** | **66.5** |
43
+
44
+ > <sup>1:只针对其中的单项选择题进行测试,即排除了填空题、开放性问题和多项选择题</sup>
45
+ > <sup>2:来源于 [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B) 的汇报结果</sup>
46
+ > <sup>3:来源于 [C-Eval](https://cevalbenchmark.com/) 的汇报结果</sup>
47
+ > <sup>4:来源于[Llama 2 论文](https://arxiv.org/abs/2307.09288)的汇报结果</sup>
48
+ >
49
+ > 对于 MMLU ,我们采用作者提供的[评测工具](https://github.com/hendrycks/test),C-Eval、AGIEval、GAOKAO-Bench、GAOKAO-English 与 MMLU 的评测方式相同,且统一采用 **5-shot** 构造测试样本。
50
+
51
+
52
+ ## Model Evaluation
53
+
54
+ In order to validate the various abilities of the model, we have chosen several comprehensive capability benchmarks across multiple disciplines, including [MMLU](https://arxiv.org/abs/2009.03300) (English), [C-Eval](https://cevalbenchmark.com/) (Chinese), [AGIEval](https://arxiv.org/abs/2304.06364) (Chinese and English), [GAOKAO-Bench](https://github.com/OpenLMLab/GAOKAO-Bench) (Chinese and English), [GAOKAO-English](https://github.com/ExpressAI/AI-Gaokao) (English), the evaluation results are as follows:
55
+
56
+ | Models\Datasets | MMLU | C-Eval | AGIEval<sup>1</sup> | GAOKAO-Bench<sup>1</sup> | GAOKAO-English<sup>1</sup> |
57
+ | :------------------------: | :--------------: | :--------------: | :-----------------: | :----------------------: | :------------------------: |
58
+ | Baichuan-13B | 51.6<sup>2</sup> | 53.6<sup>3</sup> | 40.5 | 45.9 | 56.9 |
59
+ | Llama-1-13B | 46.9<sup>4</sup> | 28.8 | 27.3 | 26.4 | 38.1 |
60
+ | Llama-2-13B | 54.8<sup>4</sup> | 35.6 | 33.4 | 35.4 | 60.6 |
61
+ | moss-moon-003-base (16B) | 24.7 | 33.1<sup>3</sup> | 26.8 | 28.5 | 34.7 |
62
+ | OpenLLaMA-13B | 42.4 | 24.7 | 24.0 | 25.6 | 33.3 |
63
+ | OPT-13B | 25.2 | 25.0 | 24.2 | 24.4 | 31.1 |
64
+ | Pythia-12B | 25.1 | 26.2 | 25.3 | 25.3 | 26.8 |
65
+ | Ziya-LLaMA-13B-Pretrain-v1 | 43.9 | 30.2 | 27.2 | 26.4 | 37.6 |
66
+ | **XVERSE-13B** | **55.1** | **54.7** | **41.4** | **53.9** | **66.5** |
67
+
68
+ > <sup>1: Tests are conducted only on single-answer multiple-choice questions, thus excluding fill-in-the-blanks, open-ended questions, and multiple-answer multiple-choice questions.</sup>
69
+ > <sup>2: Reporting results from [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B).</sup>
70
+ > <sup>3: Reporting results from [C-Eval](https://cevalbenchmark.com/).</sup>
71
+ > <sup>4: Reporting results from [Llama 2](https://arxiv.org/abs/2307.09288).</sup>
72
+ >
73
+ > For MMLU, we adopt the [evaluation tools](https://github.com/hendrycks/test) provided by the authors, C-Eval, AGIEval, GAOKAO-Bench, GAOKAO-English are the same as MMLU, and uniformly use **5-shot** to construct the test samples.
74
+
75
+ ### MMLU 各类别指标
76
+
77
+ MMLU Category Results
78
+
79
+ | 模型\类别 | Average | STEM | Social Science | Humanities | Others |
80
+ | :------------------------: | :------: | :------: | :------------: | :--------: | :------: |
81
+ | Baichuan-13B | 51.6 | 41.6 | 60.9 | 47.4 | 58.5 |
82
+ | Llama-1-13B | 46.9 | 35.8 | 53.8 | 45.0 | 53.3 |
83
+ | Llama-2-13B | 54.8 | 44.1 | 62.6 | 52.8 | 61.1 |
84
+ | moss-moon-003-base (16B) | 24.7 | 23.0 | 24.0 | 25.2 | 26.3 |
85
+ | OpenLLaMA-13B | 42.4 | 34.7 | 48.6 | 40.0 | 47.1 |
86
+ | OPT-13B | 25.2 | 23.9 | 24.1 | 25.9 | 26.3 |
87
+ | Pythia-12B | 25.1 | 24.8 | 23.0 | 26.1 | 26.0 |
88
+ | Ziya-LLaMA-13B-Pretrain-v1 | 43.9 | 36.3 | 48.8 | 41.1 | 50.3 |
89
+ | **XVERSE-13B** | **55.1** | **44.5** | **64.4** | **50.5** | **62.9** |
90
+
91
+ ### C-Eval 各类别指标
92
+
93
+ C-Eval Category Results
94
+
95
+ | 模型\类别 | Average | STEM | Social Science | Humanities | Others |
96
+ | :------------------------: | :------: | :------: | :------------: | :--------: | :------: |
97
+ | Baichuan-13B | 53.6 | 47.0 | 66.8 | 57.3 | 49.8 |
98
+ | Llama-1-13B | 28.8 | 27.5 | 33.9 | 27.7 | 27.7 |
99
+ | Llama-2-13B | 35.6 | 34.5 | 39.8 | 36.2 | 33.2 |
100
+ | moss-moon-003-base (16B) | 33.1 | 31.6 | 37.0 | 33.4 | 32.1 |
101
+ | OpenLLaMA-13B | 24.7 | 25.5 | 23.5 | 24.2 | 24.7 |
102
+ | OPT-13B | 25.0 | 24.4 | 24.6 | 25.9 | 25.4 |
103
+ | Pythia-12B | 26.2 | 26.8 | 25.1 | 26.7 | 25.4 |
104
+ | Ziya-LLaMA-13B-Pretrain-v1 | 30.2 | 27.8 | 34.3 | 32.0 | 29.0 |
105
+ | **XVERSE-13B** | **54.7** | **45.6** | **66.2** | **58.3** | **56.9** |
106
+
107
+ ### Loading with Transformers
108
+
109
+ 可通过以下代码加载 XVERSE-13B 模型进行推理:
110
+
111
+ The XVERSE-13B model can be loaded for inference using the following code:
112
+
113
+ ```python
114
+ >>> from transformers import AutoTokenizer, AutoModelForCausalLM
115
+ >>> tokenizer = AutoTokenizer.from_pretrained("xverse/XVERSE-13B")
116
+ >>> model = AutoModelForCausalLM.from_pretrained("xverse/XVERSE-13B", trust_remote_code=True).half().cuda()
117
+ >>> model = model.eval()
118
+ >>> inputs = tokenizer('北京的景点:故宫、天坛、万里长城等。\n深圳的景点:', return_tensors='pt').input_ids
119
+ >>> inputs = inputs.cuda()
120
+ >>> generated_ids = model.generate(inputs, max_new_tokens=64, eos_token_id=tokenizer.eos_token_id)
121
+ >>> print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True))
122
+ ```
123
+
124
+ 更多有关相关细节,包括文本生成demo和环境依赖,请参考我们的[Github](https://github.com/xverse-ai/XVERSE-13B)。
125
+
126
+ For more details, including the demo of text generation and environmental dependencies, please refer to our [Github](https://github.com/xverse-ai/XVERSE-13B).
127
+
128
+
129
+ ## 局限性与免责申明
130
+
131
+ XVERSE-13B 与其他所有 LLM 一样,在某些情况下可能会产生不准确、有偏见或其他令人反感的内容。因此,请谨慎使用模型生成的内容,请勿将生成的有害内容进行传播,在部署任何 XVERSE-13B 的应用之前,开发人员应根据其具体应用对模型进行安全测试和调优。
132
+
133
+ 我们强烈警告不要将 XVERSE-13B 模型用于制造或传播有害信息,或进行任何可能损害公众、国家、社会安全或违反法规的活动。如果使用 XVERSE-13B 模型产生任何问题,无论是数据安全问题、公共舆论风险,还是模型被误解、滥用、传播或不合规使用所引发的任何风险和问题,我们将不承担任何责任。
134
+
135
+ ## Limitations and Disclaimer
136
+
137
+ Like all other Large Language Models (LLMs), XVERSE-13B may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-13B, developers should conduct safety tests and optimization of the model according to its specific application.
138
+
139
+ We strongly warn against the use of the XVERSE-13B model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-13B model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.
140
+
141
+ ## 模型开源协议
142
+
143
+ 使用本仓库的源码需要遵循 [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) 开源协议,使用 XVERSE-13B 的模型权重则需要遵循[模型许可协议](MODEL_LICENSE.pdf)。
144
+
145
+ XVERSE-13B 模型权重对学术研究**完全开放**,并且支持**免费商用**,商用需申请商业使用授权,可以发送邮件到 <[email protected]> 进行申请。
146
+
147
+ ## Open Source License
148
+
149
+ The use of the source code in this repository must follow the [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) open-source license, while the use of the model weights of XVERSE-13B needs to adhere to the [Model License Agreement](MODEL_LICENSE.pdf).
150
+
151
+ The XVERSE-13B model weights are **fully open** to academic research and support **free commercial use**. Commercial use requires an application for a commercial use license by sending an email to <[email protected]>.
152
+
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "XverseForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_xverse.XverseConfig",
7
+ "AutoModelForCausalLM": "modeling_xverse.XverseForCausalLM"
8
+ },
9
+ "pad_token_id": 1,
10
+ "bos_token_id": 2,
11
+ "eos_token_id": 3,
12
+ "hidden_act": "silu",
13
+ "hidden_size": 5120,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 13824,
16
+ "max_position_embeddings": 8192,
17
+ "model_type": "xverse",
18
+ "num_attention_heads": 40,
19
+ "num_hidden_layers": 40,
20
+ "rms_norm_eps": 1e-06,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "float16",
23
+ "transformers_version": "4.28.1",
24
+ "use_cache": true,
25
+ "vocab_size": 100278
26
+ }
27
+
configuration_xverse.py ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ XVERSE model configuration"""
21
+
22
+ from transformers.configuration_utils import PretrainedConfig
23
+ from transformers.utils import logging
24
+
25
+
26
+ logger = logging.get_logger(__name__)
27
+
28
+ XVERSE_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
29
+
30
+
31
+ class XverseConfig(PretrainedConfig):
32
+ r"""
33
+ This is the configuration class to store the configuration of a [`XverseModel`]. It is used to instantiate an Xverse
34
+ model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
35
+ defaults will yield a similar configuration to that of the XVERSE-13B.
36
+
37
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
38
+ documentation from [`PretrainedConfig`] for more information.
39
+
40
+
41
+ Args:
42
+ vocab_size (`int`, *optional*, defaults to 100278):
43
+ Vocabulary size of the XVERSE model. Defines the number of different tokens that can be represented by the
44
+ `inputs_ids` passed when calling [`XverseModel`]
45
+ hidden_size (`int`, *optional*, defaults to 5120):
46
+ Dimension of the hidden representations.
47
+ intermediate_size (`int`, *optional*, defaults to 13824):
48
+ Dimension of the MLP representations.
49
+ num_hidden_layers (`int`, *optional*, defaults to 40):
50
+ Number of hidden layers in the Transformer encoder.
51
+ num_attention_heads (`int`, *optional*, defaults to 40):
52
+ Number of attention heads for each attention layer in the Transformer encoder.
53
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
54
+ The non-linear activation function (function or string) in the decoder.
55
+ max_position_embeddings (`int`, *optional*, defaults to 8192):
56
+ The maximum sequence length that this model might ever be used with. Typically set this to something large
57
+ just in case (e.g., 512 or 1024 or 2048).
58
+ initializer_range (`float`, *optional*, defaults to 0.02):
59
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
60
+ rms_norm_eps (`float`, *optional*, defaults to 1e-6):
61
+ The epsilon used by the rms normalization layers.
62
+ use_cache (`bool`, *optional*, defaults to `True`):
63
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
64
+ relevant if `config.is_decoder=True`.
65
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
66
+ Whether to tie weight embeddings
67
+
68
+ Example:
69
+
70
+ ```python
71
+ >>> from transformers import XverseModel, XverseConfig
72
+
73
+ >>> # Initializing a Xverse XVERSE-13B style configuration
74
+ >>> configuration = XverseConfig()
75
+
76
+ >>> # Initializing a model from the XVERSE-13B style configuration
77
+ >>> model = XverseModel(configuration)
78
+
79
+ >>> # Accessing the model configuration
80
+ >>> configuration = model.config
81
+ ```"""
82
+ model_type = "xverse"
83
+ keys_to_ignore_at_inference = ["past_key_values"]
84
+
85
+ def __init__(
86
+ self,
87
+ vocab_size=100278,
88
+ hidden_size=5120,
89
+ intermediate_size=13824,
90
+ num_hidden_layers=40,
91
+ num_attention_heads=40,
92
+ hidden_act="silu",
93
+ max_position_embeddings=8192,
94
+ initializer_range=0.02,
95
+ rms_norm_eps=1e-6,
96
+ use_cache=True,
97
+ pad_token_id=None,
98
+ bos_token_id=1,
99
+ eos_token_id=2,
100
+ tie_word_embeddings=False,
101
+ **kwargs,
102
+ ):
103
+ self.vocab_size = vocab_size
104
+ self.max_position_embeddings = max_position_embeddings
105
+ self.hidden_size = hidden_size
106
+ self.intermediate_size = intermediate_size
107
+ self.num_hidden_layers = num_hidden_layers
108
+ self.num_attention_heads = num_attention_heads
109
+
110
+ self.hidden_act = hidden_act
111
+ self.initializer_range = initializer_range
112
+ self.rms_norm_eps = rms_norm_eps
113
+ self.use_cache = use_cache
114
+
115
+ super().__init__(
116
+ pad_token_id=pad_token_id,
117
+ bos_token_id=bos_token_id,
118
+ eos_token_id=eos_token_id,
119
+ tie_word_embeddings=tie_word_embeddings,
120
+ **kwargs,
121
+ )
modeling_xverse.py ADDED
@@ -0,0 +1,767 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ PyTorch XVERSE model."""
21
+ import math
22
+ from typing import List, Optional, Tuple, Union
23
+
24
+ import torch
25
+ import torch.nn.functional as F
26
+ import torch.utils.checkpoint
27
+ from torch import nn
28
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
29
+
30
+ from transformers.activations import ACT2FN
31
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
32
+ from transformers.modeling_utils import PreTrainedModel
33
+ from transformers.utils import add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings
34
+ from .configuration_xverse import XverseConfig
35
+
36
+
37
+ logger = logging.get_logger(__name__)
38
+
39
+ _CONFIG_FOR_DOC = "XverseConfig"
40
+
41
+
42
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
43
+ def _make_causal_mask(
44
+ input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
45
+ ):
46
+ """
47
+ Make causal mask used for bi-directional self-attention.
48
+ """
49
+ bsz, tgt_len = input_ids_shape
50
+ mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device)
51
+ mask_cond = torch.arange(mask.size(-1), device=device)
52
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
53
+ mask = mask.to(dtype)
54
+
55
+ if past_key_values_length > 0:
56
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
57
+ return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
58
+
59
+
60
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
61
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
62
+ """
63
+ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
64
+ """
65
+ bsz, src_len = mask.size()
66
+ tgt_len = tgt_len if tgt_len is not None else src_len
67
+
68
+ expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
69
+
70
+ inverted_mask = 1.0 - expanded_mask
71
+
72
+ return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
73
+
74
+
75
+ class XverseRMSNorm(nn.Module):
76
+ def __init__(self, hidden_size, eps=1e-6):
77
+ """
78
+ XverseRMSNorm is equivalent to T5LayerNorm
79
+ """
80
+ super().__init__()
81
+ self.weight = nn.Parameter(torch.ones(hidden_size))
82
+ self.variance_epsilon = eps
83
+
84
+ def forward(self, hidden_states):
85
+ input_dtype = hidden_states.dtype
86
+ variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
87
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
88
+
89
+ return (self.weight * hidden_states).to(input_dtype)
90
+
91
+
92
+ class XverseRotaryEmbedding(torch.nn.Module):
93
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
94
+ super().__init__()
95
+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float().to(device) / dim))
96
+ self.register_buffer("inv_freq", inv_freq)
97
+
98
+ # Build here to make `torch.jit.trace` work.
99
+ self.max_seq_len_cached = max_position_embeddings
100
+ t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
101
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
102
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
103
+ emb = torch.cat((freqs, freqs), dim=-1)
104
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :], persistent=False)
105
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :], persistent=False)
106
+
107
+ def forward(self, x, seq_len=None):
108
+ # x: [bs, num_attention_heads, seq_len, head_size]
109
+ # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case.
110
+ if seq_len > self.max_seq_len_cached:
111
+ self.max_seq_len_cached = seq_len
112
+ t = torch.arange(self.max_seq_len_cached, device=x.device, dtype=self.inv_freq.dtype)
113
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
114
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
115
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
116
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :], persistent=False)
117
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :], persistent=False)
118
+ return (
119
+ self.cos_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
120
+ self.sin_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
121
+ )
122
+
123
+
124
+ def rotate_half(x):
125
+ """Rotates half the hidden dims of the input."""
126
+ x1 = x[..., : x.shape[-1] // 2]
127
+ x2 = x[..., x.shape[-1] // 2 :]
128
+ return torch.cat((-x2, x1), dim=-1)
129
+
130
+
131
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
132
+ # The first two dimensions of cos and sin are always 1, so we can `squeeze` them.
133
+ cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
134
+ sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
135
+ cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
136
+ sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
137
+ q_embed = (q * cos) + (rotate_half(q) * sin)
138
+ k_embed = (k * cos) + (rotate_half(k) * sin)
139
+ return q_embed, k_embed
140
+
141
+
142
+ class XverseMLP(nn.Module):
143
+ def __init__(
144
+ self,
145
+ hidden_size: int,
146
+ intermediate_size: int,
147
+ hidden_act: str,
148
+ ):
149
+ super().__init__()
150
+ self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
151
+ self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
152
+ self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
153
+ self.act_fn = ACT2FN[hidden_act]
154
+
155
+ def forward(self, x):
156
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
157
+
158
+
159
+ class XverseAttention(nn.Module):
160
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
161
+
162
+ def __init__(self, config: XverseConfig):
163
+ super().__init__()
164
+ self.config = config
165
+ self.hidden_size = config.hidden_size
166
+ self.num_heads = config.num_attention_heads
167
+ self.head_dim = self.hidden_size // self.num_heads
168
+ self.max_position_embeddings = config.max_position_embeddings
169
+
170
+ if (self.head_dim * self.num_heads) != self.hidden_size:
171
+ raise ValueError(
172
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
173
+ f" and `num_heads`: {self.num_heads})."
174
+ )
175
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
176
+ self.k_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
177
+ self.v_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
178
+ self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)
179
+ self.rotary_emb = XverseRotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings)
180
+
181
+ def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
182
+ return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
183
+
184
+ def forward(
185
+ self,
186
+ hidden_states: torch.Tensor,
187
+ attention_mask: Optional[torch.Tensor] = None,
188
+ position_ids: Optional[torch.LongTensor] = None,
189
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
190
+ output_attentions: bool = False,
191
+ use_cache: bool = False,
192
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
193
+ bsz, q_len, _ = hidden_states.size()
194
+
195
+ query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
196
+ key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
197
+ value_states = self.v_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
198
+
199
+ kv_seq_len = key_states.shape[-2]
200
+ if past_key_value is not None:
201
+ kv_seq_len += past_key_value[0].shape[-2]
202
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
203
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
204
+ # [bsz, nh, t, hd]
205
+
206
+ if past_key_value is not None:
207
+ # reuse k, v, self_attention
208
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
209
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
210
+
211
+ past_key_value = (key_states, value_states) if use_cache else None
212
+
213
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
214
+
215
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
216
+ raise ValueError(
217
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
218
+ f" {attn_weights.size()}"
219
+ )
220
+
221
+ if attention_mask is not None:
222
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
223
+ raise ValueError(
224
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
225
+ )
226
+ attn_weights = attn_weights + attention_mask
227
+ attn_weights = torch.max(
228
+ attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min, device=attn_weights.device)
229
+ )
230
+
231
+ # upcast attention to fp32
232
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
233
+ attn_output = torch.matmul(attn_weights, value_states)
234
+
235
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
236
+ raise ValueError(
237
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
238
+ f" {attn_output.size()}"
239
+ )
240
+
241
+ attn_output = attn_output.transpose(1, 2)
242
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
243
+
244
+ attn_output = self.o_proj(attn_output)
245
+
246
+ if not output_attentions:
247
+ attn_weights = None
248
+
249
+ return attn_output, attn_weights, past_key_value
250
+
251
+
252
+ class XverseDecoderLayer(nn.Module):
253
+ def __init__(self, config: XverseConfig):
254
+ super().__init__()
255
+ self.hidden_size = config.hidden_size
256
+ self.self_attn = XverseAttention(config=config)
257
+ self.mlp = XverseMLP(
258
+ hidden_size=self.hidden_size,
259
+ intermediate_size=config.intermediate_size,
260
+ hidden_act=config.hidden_act,
261
+ )
262
+ self.input_layernorm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
263
+ self.post_attention_layernorm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
264
+
265
+ def forward(
266
+ self,
267
+ hidden_states: torch.Tensor,
268
+ attention_mask: Optional[torch.Tensor] = None,
269
+ position_ids: Optional[torch.LongTensor] = None,
270
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
271
+ output_attentions: Optional[bool] = False,
272
+ use_cache: Optional[bool] = False,
273
+ ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
274
+ """
275
+ Args:
276
+ hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
277
+ attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
278
+ `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
279
+ output_attentions (`bool`, *optional*):
280
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under
281
+ returned tensors for more detail.
282
+ use_cache (`bool`, *optional*):
283
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
284
+ (see `past_key_values`).
285
+ past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
286
+ """
287
+
288
+ residual = hidden_states
289
+
290
+ hidden_states = self.input_layernorm(hidden_states)
291
+
292
+ # Self Attention
293
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
294
+ hidden_states=hidden_states,
295
+ attention_mask=attention_mask,
296
+ position_ids=position_ids,
297
+ past_key_value=past_key_value,
298
+ output_attentions=output_attentions,
299
+ use_cache=use_cache,
300
+ )
301
+ hidden_states = residual + hidden_states
302
+
303
+ # Fully Connected
304
+ residual = hidden_states
305
+ hidden_states = self.post_attention_layernorm(hidden_states)
306
+ hidden_states = self.mlp(hidden_states)
307
+ hidden_states = residual + hidden_states
308
+
309
+ outputs = (hidden_states,)
310
+
311
+ if output_attentions:
312
+ outputs += (self_attn_weights,)
313
+
314
+ if use_cache:
315
+ outputs += (present_key_value,)
316
+
317
+ return outputs
318
+
319
+
320
+ XVERSE_START_DOCSTRING = r"""
321
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
322
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
323
+ etc.)
324
+
325
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
326
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
327
+ and behavior.
328
+
329
+ Parameters:
330
+ config ([`XverseConfig`]):
331
+ Model configuration class with all the parameters of the model. Initializing with a config file does not
332
+ load the weights associated with the model, only the configuration. Check out the
333
+ [`~PreTrainedModel.from_pretrained`] method to load the model weights.
334
+ """
335
+
336
+
337
+ @add_start_docstrings(
338
+ "The bare Xverse Model outputting raw hidden-states without any specific head on top.",
339
+ XVERSE_START_DOCSTRING,
340
+ )
341
+ class XversePreTrainedModel(PreTrainedModel):
342
+ config_class = XverseConfig
343
+ base_model_prefix = "model"
344
+ supports_gradient_checkpointing = True
345
+ _no_split_modules = ["XverseDecoderLayer"]
346
+ _skip_keys_device_placement = "past_key_values"
347
+ _keys_to_ignore_on_load_unexpected = [r"decoder\.version"]
348
+
349
+ def _init_weights(self, module):
350
+ std = self.config.initializer_range
351
+ if isinstance(module, nn.Linear):
352
+ module.weight.data.normal_(mean=0.0, std=std)
353
+ if module.bias is not None:
354
+ module.bias.data.zero_()
355
+ elif isinstance(module, nn.Embedding):
356
+ module.weight.data.normal_(mean=0.0, std=std)
357
+ if module.padding_idx is not None:
358
+ module.weight.data[module.padding_idx].zero_()
359
+
360
+ def _set_gradient_checkpointing(self, module, value=False):
361
+ if isinstance(module, XverseModel):
362
+ module.gradient_checkpointing = value
363
+
364
+
365
+ XVERSE_INPUTS_DOCSTRING = r"""
366
+ Args:
367
+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
368
+ Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
369
+ it.
370
+
371
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
372
+ [`PreTrainedTokenizer.__call__`] for details.
373
+
374
+ [What are input IDs?](../glossary#input-ids)
375
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
376
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
377
+
378
+ - 1 for tokens that are **not masked**,
379
+ - 0 for tokens that are **masked**.
380
+
381
+ [What are attention masks?](../glossary#attention-mask)
382
+
383
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
384
+ [`PreTrainedTokenizer.__call__`] for details.
385
+
386
+ If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see
387
+ `past_key_values`).
388
+
389
+ If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
390
+ and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
391
+ information on the default strategy.
392
+
393
+ - 1 indicates the head is **not masked**,
394
+ - 0 indicates the head is **masked**.
395
+ position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
396
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
397
+ config.n_positions - 1]`.
398
+
399
+ [What are position IDs?](../glossary#position-ids)
400
+ past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
401
+ Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape
402
+ `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape
403
+ `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.
404
+
405
+ Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
406
+ blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
407
+
408
+ If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
409
+ don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
410
+ `decoder_input_ids` of shape `(batch_size, sequence_length)`.
411
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
412
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
413
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
414
+ model's internal embedding lookup matrix.
415
+ use_cache (`bool`, *optional*):
416
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
417
+ `past_key_values`).
418
+ output_attentions (`bool`, *optional*):
419
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
420
+ tensors for more detail.
421
+ output_hidden_states (`bool`, *optional*):
422
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
423
+ more detail.
424
+ return_dict (`bool`, *optional*):
425
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
426
+ """
427
+
428
+ @add_start_docstrings(
429
+ "The bare Xverse Model outputting raw hidden-states without any specific head on top.",
430
+ XVERSE_START_DOCSTRING,
431
+ )
432
+ class XverseModel(XversePreTrainedModel):
433
+ """
434
+ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`XverseDecoderLayer`]
435
+
436
+ Args:
437
+ config: XverseConfig
438
+ """
439
+
440
+ def __init__(self, config: XverseConfig):
441
+ super().__init__(config)
442
+ self.padding_idx = config.pad_token_id
443
+ self.vocab_size = config.vocab_size
444
+
445
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
446
+ self.layers = nn.ModuleList([XverseDecoderLayer(config) for _ in range(config.num_hidden_layers)])
447
+ self.norm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
448
+
449
+ self.gradient_checkpointing = False
450
+ # Initialize weights and apply final processing
451
+ self.post_init()
452
+
453
+ def get_input_embeddings(self):
454
+ return self.embed_tokens
455
+
456
+ def set_input_embeddings(self, value):
457
+ self.embed_tokens = value
458
+
459
+ # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
460
+ def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
461
+ # create causal mask
462
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
463
+ combined_attention_mask = None
464
+ if input_shape[-1] > 1:
465
+ combined_attention_mask = _make_causal_mask(
466
+ input_shape,
467
+ inputs_embeds.dtype,
468
+ device=inputs_embeds.device,
469
+ past_key_values_length=past_key_values_length,
470
+ )
471
+
472
+ if attention_mask is not None:
473
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
474
+ expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
475
+ inputs_embeds.device
476
+ )
477
+ combined_attention_mask = (
478
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
479
+ )
480
+
481
+ return combined_attention_mask
482
+
483
+ @add_start_docstrings_to_model_forward(XVERSE_INPUTS_DOCSTRING)
484
+ def forward(
485
+ self,
486
+ input_ids: torch.LongTensor = None,
487
+ attention_mask: Optional[torch.Tensor] = None,
488
+ position_ids: Optional[torch.LongTensor] = None,
489
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
490
+ inputs_embeds: Optional[torch.FloatTensor] = None,
491
+ use_cache: Optional[bool] = None,
492
+ output_attentions: Optional[bool] = None,
493
+ output_hidden_states: Optional[bool] = None,
494
+ return_dict: Optional[bool] = None,
495
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
496
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
497
+ output_hidden_states = (
498
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
499
+ )
500
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
501
+
502
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
503
+
504
+ # retrieve input_ids and inputs_embeds
505
+ if input_ids is not None and inputs_embeds is not None:
506
+ raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
507
+ elif input_ids is not None:
508
+ batch_size, seq_length = input_ids.shape
509
+ elif inputs_embeds is not None:
510
+ batch_size, seq_length, _ = inputs_embeds.shape
511
+ else:
512
+ raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
513
+
514
+ seq_length_with_past = seq_length
515
+ past_key_values_length = 0
516
+
517
+ if past_key_values is not None:
518
+ past_key_values_length = past_key_values[0][0].shape[2]
519
+ seq_length_with_past = seq_length_with_past + past_key_values_length
520
+
521
+ if position_ids is None:
522
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
523
+ position_ids = torch.arange(
524
+ past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
525
+ )
526
+ position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
527
+ else:
528
+ position_ids = position_ids.view(-1, seq_length).long()
529
+
530
+ if inputs_embeds is None:
531
+ inputs_embeds = self.embed_tokens(input_ids)
532
+ # embed positions
533
+ if attention_mask is None:
534
+ attention_mask = torch.ones(
535
+ (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device
536
+ )
537
+ attention_mask = self._prepare_decoder_attention_mask(
538
+ attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length
539
+ )
540
+
541
+ hidden_states = inputs_embeds
542
+
543
+ if self.gradient_checkpointing and self.training:
544
+ if use_cache:
545
+ logger.warning_once(
546
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
547
+ )
548
+ use_cache = False
549
+
550
+ # decoder layers
551
+ all_hidden_states = () if output_hidden_states else None
552
+ all_self_attns = () if output_attentions else None
553
+ next_decoder_cache = () if use_cache else None
554
+
555
+ for idx, decoder_layer in enumerate(self.layers):
556
+ if output_hidden_states:
557
+ all_hidden_states += (hidden_states,)
558
+
559
+ past_key_value = past_key_values[idx] if past_key_values is not None else None
560
+
561
+ if self.gradient_checkpointing and self.training:
562
+
563
+ def create_custom_forward(module):
564
+ def custom_forward(*inputs):
565
+ # None for past_key_value
566
+ return module(*inputs, output_attentions, None)
567
+
568
+ return custom_forward
569
+
570
+ layer_outputs = torch.utils.checkpoint.checkpoint(
571
+ create_custom_forward(decoder_layer),
572
+ hidden_states,
573
+ attention_mask,
574
+ position_ids,
575
+ None,
576
+ )
577
+ else:
578
+ layer_outputs = decoder_layer(
579
+ hidden_states,
580
+ attention_mask=attention_mask,
581
+ position_ids=position_ids,
582
+ past_key_value=past_key_value,
583
+ output_attentions=output_attentions,
584
+ use_cache=use_cache,
585
+ )
586
+
587
+ hidden_states = layer_outputs[0]
588
+
589
+ if use_cache:
590
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
591
+
592
+ if output_attentions:
593
+ all_self_attns += (layer_outputs[1],)
594
+
595
+ hidden_states = self.norm(hidden_states)
596
+
597
+ # add hidden states from the last decoder layer
598
+ if output_hidden_states:
599
+ all_hidden_states += (hidden_states,)
600
+
601
+ next_cache = next_decoder_cache if use_cache else None
602
+ if not return_dict:
603
+ return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
604
+ return BaseModelOutputWithPast(
605
+ last_hidden_state=hidden_states,
606
+ past_key_values=next_cache,
607
+ hidden_states=all_hidden_states,
608
+ attentions=all_self_attns,
609
+ )
610
+
611
+
612
+ class XverseForCausalLM(XversePreTrainedModel):
613
+ _tied_weights_keys = ["lm_head.weight"]
614
+
615
+ def __init__(self, config):
616
+ super().__init__(config)
617
+ self.model = XverseModel(config)
618
+
619
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
620
+
621
+ # Initialize weights and apply final processing
622
+ self.post_init()
623
+
624
+ def get_input_embeddings(self):
625
+ return self.model.embed_tokens
626
+
627
+ def set_input_embeddings(self, value):
628
+ self.model.embed_tokens = value
629
+
630
+ def get_output_embeddings(self):
631
+ return self.lm_head
632
+
633
+ def set_output_embeddings(self, new_embeddings):
634
+ self.lm_head = new_embeddings
635
+
636
+ def set_decoder(self, decoder):
637
+ self.model = decoder
638
+
639
+ def get_decoder(self):
640
+ return self.model
641
+
642
+ @add_start_docstrings_to_model_forward(XVERSE_INPUTS_DOCSTRING)
643
+ @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
644
+ def forward(
645
+ self,
646
+ input_ids: torch.LongTensor = None,
647
+ attention_mask: Optional[torch.Tensor] = None,
648
+ position_ids: Optional[torch.LongTensor] = None,
649
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
650
+ inputs_embeds: Optional[torch.FloatTensor] = None,
651
+ labels: Optional[torch.LongTensor] = None,
652
+ use_cache: Optional[bool] = None,
653
+ output_attentions: Optional[bool] = None,
654
+ output_hidden_states: Optional[bool] = None,
655
+ return_dict: Optional[bool] = None,
656
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
657
+ r"""
658
+ Args:
659
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
660
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
661
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
662
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
663
+
664
+ Returns:
665
+
666
+ Example:
667
+
668
+ ```python
669
+ >>> from transformers import AutoTokenizer, AutoModelForCausalLM
670
+
671
+ >>> model = AutoModelForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS, trust_remote_code=True)
672
+ >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
673
+
674
+ >>> prompt = "Hey, are you conscious? Can you talk to me?"
675
+ >>> inputs = tokenizer(prompt, return_tensors="pt")
676
+
677
+ >>> # Generate
678
+ >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
679
+ >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
680
+ "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
681
+ ```"""
682
+
683
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
684
+ output_hidden_states = (
685
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
686
+ )
687
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
688
+
689
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
690
+ outputs = self.model(
691
+ input_ids=input_ids,
692
+ attention_mask=attention_mask,
693
+ position_ids=position_ids,
694
+ past_key_values=past_key_values,
695
+ inputs_embeds=inputs_embeds,
696
+ use_cache=use_cache,
697
+ output_attentions=output_attentions,
698
+ output_hidden_states=output_hidden_states,
699
+ return_dict=return_dict,
700
+ )
701
+
702
+ hidden_states = outputs[0]
703
+ logits = self.lm_head(hidden_states)
704
+
705
+ loss = None
706
+ if labels is not None:
707
+ # Shift so that tokens < n predict n
708
+ shift_logits = logits[..., :-1, :].contiguous()
709
+ shift_labels = labels[..., 1:].contiguous()
710
+ # Flatten the tokens
711
+ loss_fct = CrossEntropyLoss()
712
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
713
+ shift_labels = shift_labels.view(-1)
714
+ # Enable model parallelism
715
+ shift_labels = shift_labels.to(shift_logits.device)
716
+ loss = loss_fct(shift_logits, shift_labels)
717
+
718
+ if not return_dict:
719
+ output = (logits,) + outputs[1:]
720
+ return (loss,) + output if loss is not None else output
721
+
722
+ return CausalLMOutputWithPast(
723
+ loss=loss,
724
+ logits=logits,
725
+ past_key_values=outputs.past_key_values,
726
+ hidden_states=outputs.hidden_states,
727
+ attentions=outputs.attentions,
728
+ )
729
+
730
+ def prepare_inputs_for_generation(
731
+ self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
732
+ ):
733
+ if past_key_values:
734
+ input_ids = input_ids[:, -1:]
735
+
736
+ position_ids = kwargs.get("position_ids", None)
737
+ if attention_mask is not None and position_ids is None:
738
+ # create position_ids on the fly for batch generation
739
+ position_ids = attention_mask.long().cumsum(-1) - 1
740
+ position_ids.masked_fill_(attention_mask == 0, 1)
741
+ if past_key_values:
742
+ position_ids = position_ids[:, -1].unsqueeze(-1)
743
+
744
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
745
+ if inputs_embeds is not None and past_key_values is None:
746
+ model_inputs = {"inputs_embeds": inputs_embeds}
747
+ else:
748
+ model_inputs = {"input_ids": input_ids}
749
+
750
+ model_inputs.update(
751
+ {
752
+ "position_ids": position_ids,
753
+ "past_key_values": past_key_values,
754
+ "use_cache": kwargs.get("use_cache"),
755
+ "attention_mask": attention_mask,
756
+ }
757
+ )
758
+ return model_inputs
759
+
760
+ @staticmethod
761
+ def _reorder_cache(past_key_values, beam_idx):
762
+ reordered_past = ()
763
+ for layer_past in past_key_values:
764
+ reordered_past += (
765
+ tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
766
+ )
767
+ return reordered_past
pytorch_model-00001-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18696855f373f92e774ac4ae12d80fe66b1c1e9acc7b98cd1029fef9e9d228d3
3
+ size 10065941445
pytorch_model-00002-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:10bb394d79391a86c887edce8e7d2f1b921c60fcd9aefe89ee5fe6de9b370a0f
3
+ size 10009010414
pytorch_model-00003-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:defb5f082a0b8924456672401a1d691cae47ce0e01d20e07255c8718d0d7b955
3
+ size 7512554533
pytorch_model.bin.index.json ADDED
@@ -0,0 +1,410 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 27430067200
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "pytorch_model-00003-of-00003.bin",
7
+ "model.embed_tokens.weight": "pytorch_model-00001-of-00003.bin",
8
+ "model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
9
+ "model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
10
+ "model.layers.0.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
11
+ "model.layers.0.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
12
+ "model.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
13
+ "model.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
14
+ "model.layers.0.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
15
+ "model.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
16
+ "model.layers.0.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
17
+ "model.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
18
+ "model.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
19
+ "model.layers.1.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
20
+ "model.layers.1.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
21
+ "model.layers.1.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
22
+ "model.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
23
+ "model.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
24
+ "model.layers.1.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
25
+ "model.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
26
+ "model.layers.1.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
27
+ "model.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
28
+ "model.layers.10.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
29
+ "model.layers.10.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
30
+ "model.layers.10.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
31
+ "model.layers.10.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
32
+ "model.layers.10.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
33
+ "model.layers.10.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
34
+ "model.layers.10.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
35
+ "model.layers.10.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
36
+ "model.layers.10.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
37
+ "model.layers.10.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
38
+ "model.layers.11.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
39
+ "model.layers.11.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
40
+ "model.layers.11.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
41
+ "model.layers.11.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
42
+ "model.layers.11.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
43
+ "model.layers.11.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
44
+ "model.layers.11.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
45
+ "model.layers.11.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
46
+ "model.layers.11.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
47
+ "model.layers.11.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
48
+ "model.layers.12.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
49
+ "model.layers.12.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
50
+ "model.layers.12.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
51
+ "model.layers.12.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
52
+ "model.layers.12.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
53
+ "model.layers.12.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
54
+ "model.layers.12.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
55
+ "model.layers.12.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
56
+ "model.layers.12.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
57
+ "model.layers.12.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
58
+ "model.layers.13.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
59
+ "model.layers.13.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
60
+ "model.layers.13.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
61
+ "model.layers.13.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
62
+ "model.layers.13.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
63
+ "model.layers.13.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
64
+ "model.layers.13.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
65
+ "model.layers.13.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
66
+ "model.layers.13.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
67
+ "model.layers.13.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
68
+ "model.layers.14.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
69
+ "model.layers.14.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
70
+ "model.layers.14.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
71
+ "model.layers.14.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
72
+ "model.layers.14.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
73
+ "model.layers.14.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
74
+ "model.layers.14.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
75
+ "model.layers.14.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
76
+ "model.layers.14.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
77
+ "model.layers.14.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
78
+ "model.layers.15.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
79
+ "model.layers.15.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
80
+ "model.layers.15.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
81
+ "model.layers.15.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
82
+ "model.layers.15.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
83
+ "model.layers.15.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
84
+ "model.layers.15.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
85
+ "model.layers.15.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
86
+ "model.layers.15.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
87
+ "model.layers.15.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
88
+ "model.layers.16.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
89
+ "model.layers.16.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
90
+ "model.layers.16.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
91
+ "model.layers.16.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
92
+ "model.layers.16.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
93
+ "model.layers.16.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
94
+ "model.layers.16.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
95
+ "model.layers.16.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
96
+ "model.layers.16.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
97
+ "model.layers.16.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
98
+ "model.layers.17.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
99
+ "model.layers.17.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
100
+ "model.layers.17.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
101
+ "model.layers.17.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
102
+ "model.layers.17.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
103
+ "model.layers.17.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
104
+ "model.layers.17.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
105
+ "model.layers.17.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
106
+ "model.layers.17.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
107
+ "model.layers.17.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
108
+ "model.layers.18.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
109
+ "model.layers.18.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
110
+ "model.layers.18.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
111
+ "model.layers.18.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
112
+ "model.layers.18.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
113
+ "model.layers.18.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
114
+ "model.layers.18.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
115
+ "model.layers.18.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
116
+ "model.layers.18.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
117
+ "model.layers.18.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
118
+ "model.layers.19.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
119
+ "model.layers.19.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
120
+ "model.layers.19.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
121
+ "model.layers.19.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
122
+ "model.layers.19.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
123
+ "model.layers.19.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
124
+ "model.layers.19.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
125
+ "model.layers.19.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
126
+ "model.layers.19.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
127
+ "model.layers.19.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
128
+ "model.layers.2.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
129
+ "model.layers.2.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
130
+ "model.layers.2.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
131
+ "model.layers.2.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
132
+ "model.layers.2.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
133
+ "model.layers.2.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
134
+ "model.layers.2.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
135
+ "model.layers.2.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
136
+ "model.layers.2.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
137
+ "model.layers.2.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
138
+ "model.layers.20.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
139
+ "model.layers.20.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
140
+ "model.layers.20.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
141
+ "model.layers.20.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
142
+ "model.layers.20.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
143
+ "model.layers.20.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
144
+ "model.layers.20.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
145
+ "model.layers.20.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
146
+ "model.layers.20.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
147
+ "model.layers.20.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
148
+ "model.layers.21.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
149
+ "model.layers.21.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
150
+ "model.layers.21.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
151
+ "model.layers.21.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
152
+ "model.layers.21.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
153
+ "model.layers.21.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
154
+ "model.layers.21.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
155
+ "model.layers.21.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
156
+ "model.layers.21.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
157
+ "model.layers.21.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
158
+ "model.layers.22.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
159
+ "model.layers.22.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
160
+ "model.layers.22.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
161
+ "model.layers.22.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
162
+ "model.layers.22.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
163
+ "model.layers.22.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
164
+ "model.layers.22.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
165
+ "model.layers.22.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
166
+ "model.layers.22.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
167
+ "model.layers.22.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
168
+ "model.layers.23.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
169
+ "model.layers.23.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
170
+ "model.layers.23.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
171
+ "model.layers.23.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
172
+ "model.layers.23.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
173
+ "model.layers.23.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
174
+ "model.layers.23.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
175
+ "model.layers.23.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
176
+ "model.layers.23.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
177
+ "model.layers.23.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
178
+ "model.layers.24.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
179
+ "model.layers.24.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
180
+ "model.layers.24.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
181
+ "model.layers.24.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
182
+ "model.layers.24.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
183
+ "model.layers.24.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
184
+ "model.layers.24.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
185
+ "model.layers.24.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
186
+ "model.layers.24.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
187
+ "model.layers.24.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
188
+ "model.layers.25.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
189
+ "model.layers.25.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
190
+ "model.layers.25.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
191
+ "model.layers.25.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
192
+ "model.layers.25.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
193
+ "model.layers.25.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
194
+ "model.layers.25.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
195
+ "model.layers.25.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
196
+ "model.layers.25.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
197
+ "model.layers.25.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
198
+ "model.layers.26.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
199
+ "model.layers.26.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
200
+ "model.layers.26.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
201
+ "model.layers.26.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
202
+ "model.layers.26.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
203
+ "model.layers.26.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
204
+ "model.layers.26.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
205
+ "model.layers.26.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
206
+ "model.layers.26.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
207
+ "model.layers.26.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
208
+ "model.layers.27.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
209
+ "model.layers.27.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
210
+ "model.layers.27.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
211
+ "model.layers.27.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
212
+ "model.layers.27.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
213
+ "model.layers.27.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
214
+ "model.layers.27.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
215
+ "model.layers.27.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
216
+ "model.layers.27.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
217
+ "model.layers.27.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
218
+ "model.layers.28.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
219
+ "model.layers.28.mlp.down_proj.weight": "pytorch_model-00002-of-00003.bin",
220
+ "model.layers.28.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
221
+ "model.layers.28.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
222
+ "model.layers.28.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
223
+ "model.layers.28.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
224
+ "model.layers.28.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
225
+ "model.layers.28.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
226
+ "model.layers.28.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
227
+ "model.layers.28.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
228
+ "model.layers.29.input_layernorm.weight": "pytorch_model-00002-of-00003.bin",
229
+ "model.layers.29.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
230
+ "model.layers.29.mlp.gate_proj.weight": "pytorch_model-00002-of-00003.bin",
231
+ "model.layers.29.mlp.up_proj.weight": "pytorch_model-00002-of-00003.bin",
232
+ "model.layers.29.post_attention_layernorm.weight": "pytorch_model-00002-of-00003.bin",
233
+ "model.layers.29.self_attn.k_proj.weight": "pytorch_model-00002-of-00003.bin",
234
+ "model.layers.29.self_attn.o_proj.weight": "pytorch_model-00002-of-00003.bin",
235
+ "model.layers.29.self_attn.q_proj.weight": "pytorch_model-00002-of-00003.bin",
236
+ "model.layers.29.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00003.bin",
237
+ "model.layers.29.self_attn.v_proj.weight": "pytorch_model-00002-of-00003.bin",
238
+ "model.layers.3.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
239
+ "model.layers.3.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
240
+ "model.layers.3.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
241
+ "model.layers.3.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
242
+ "model.layers.3.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
243
+ "model.layers.3.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
244
+ "model.layers.3.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
245
+ "model.layers.3.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
246
+ "model.layers.3.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
247
+ "model.layers.3.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
248
+ "model.layers.30.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
249
+ "model.layers.30.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
250
+ "model.layers.30.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
251
+ "model.layers.30.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
252
+ "model.layers.30.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
253
+ "model.layers.30.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
254
+ "model.layers.30.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
255
+ "model.layers.30.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
256
+ "model.layers.30.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
257
+ "model.layers.30.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
258
+ "model.layers.31.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
259
+ "model.layers.31.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
260
+ "model.layers.31.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
261
+ "model.layers.31.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
262
+ "model.layers.31.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
263
+ "model.layers.31.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
264
+ "model.layers.31.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
265
+ "model.layers.31.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
266
+ "model.layers.31.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
267
+ "model.layers.31.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
268
+ "model.layers.32.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
269
+ "model.layers.32.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
270
+ "model.layers.32.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
271
+ "model.layers.32.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
272
+ "model.layers.32.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
273
+ "model.layers.32.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
274
+ "model.layers.32.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
275
+ "model.layers.32.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
276
+ "model.layers.32.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
277
+ "model.layers.32.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
278
+ "model.layers.33.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
279
+ "model.layers.33.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
280
+ "model.layers.33.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
281
+ "model.layers.33.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
282
+ "model.layers.33.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
283
+ "model.layers.33.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
284
+ "model.layers.33.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
285
+ "model.layers.33.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
286
+ "model.layers.33.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
287
+ "model.layers.33.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
288
+ "model.layers.34.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
289
+ "model.layers.34.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
290
+ "model.layers.34.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
291
+ "model.layers.34.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
292
+ "model.layers.34.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
293
+ "model.layers.34.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
294
+ "model.layers.34.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
295
+ "model.layers.34.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
296
+ "model.layers.34.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
297
+ "model.layers.34.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
298
+ "model.layers.35.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
299
+ "model.layers.35.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
300
+ "model.layers.35.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
301
+ "model.layers.35.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
302
+ "model.layers.35.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
303
+ "model.layers.35.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
304
+ "model.layers.35.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
305
+ "model.layers.35.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
306
+ "model.layers.35.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
307
+ "model.layers.35.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
308
+ "model.layers.36.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
309
+ "model.layers.36.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
310
+ "model.layers.36.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
311
+ "model.layers.36.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
312
+ "model.layers.36.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
313
+ "model.layers.36.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
314
+ "model.layers.36.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
315
+ "model.layers.36.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
316
+ "model.layers.36.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
317
+ "model.layers.36.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
318
+ "model.layers.37.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
319
+ "model.layers.37.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
320
+ "model.layers.37.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
321
+ "model.layers.37.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
322
+ "model.layers.37.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
323
+ "model.layers.37.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
324
+ "model.layers.37.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
325
+ "model.layers.37.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
326
+ "model.layers.37.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
327
+ "model.layers.37.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
328
+ "model.layers.38.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
329
+ "model.layers.38.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
330
+ "model.layers.38.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
331
+ "model.layers.38.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
332
+ "model.layers.38.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
333
+ "model.layers.38.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
334
+ "model.layers.38.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
335
+ "model.layers.38.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
336
+ "model.layers.38.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
337
+ "model.layers.38.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
338
+ "model.layers.39.input_layernorm.weight": "pytorch_model-00003-of-00003.bin",
339
+ "model.layers.39.mlp.down_proj.weight": "pytorch_model-00003-of-00003.bin",
340
+ "model.layers.39.mlp.gate_proj.weight": "pytorch_model-00003-of-00003.bin",
341
+ "model.layers.39.mlp.up_proj.weight": "pytorch_model-00003-of-00003.bin",
342
+ "model.layers.39.post_attention_layernorm.weight": "pytorch_model-00003-of-00003.bin",
343
+ "model.layers.39.self_attn.k_proj.weight": "pytorch_model-00003-of-00003.bin",
344
+ "model.layers.39.self_attn.o_proj.weight": "pytorch_model-00003-of-00003.bin",
345
+ "model.layers.39.self_attn.q_proj.weight": "pytorch_model-00003-of-00003.bin",
346
+ "model.layers.39.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00003.bin",
347
+ "model.layers.39.self_attn.v_proj.weight": "pytorch_model-00003-of-00003.bin",
348
+ "model.layers.4.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
349
+ "model.layers.4.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
350
+ "model.layers.4.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
351
+ "model.layers.4.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
352
+ "model.layers.4.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
353
+ "model.layers.4.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
354
+ "model.layers.4.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
355
+ "model.layers.4.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
356
+ "model.layers.4.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
357
+ "model.layers.4.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
358
+ "model.layers.5.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
359
+ "model.layers.5.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
360
+ "model.layers.5.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
361
+ "model.layers.5.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
362
+ "model.layers.5.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
363
+ "model.layers.5.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
364
+ "model.layers.5.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
365
+ "model.layers.5.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
366
+ "model.layers.5.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
367
+ "model.layers.5.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
368
+ "model.layers.6.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
369
+ "model.layers.6.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
370
+ "model.layers.6.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
371
+ "model.layers.6.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
372
+ "model.layers.6.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
373
+ "model.layers.6.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
374
+ "model.layers.6.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
375
+ "model.layers.6.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
376
+ "model.layers.6.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
377
+ "model.layers.6.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
378
+ "model.layers.7.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
379
+ "model.layers.7.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
380
+ "model.layers.7.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
381
+ "model.layers.7.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
382
+ "model.layers.7.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
383
+ "model.layers.7.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
384
+ "model.layers.7.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
385
+ "model.layers.7.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
386
+ "model.layers.7.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
387
+ "model.layers.7.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
388
+ "model.layers.8.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
389
+ "model.layers.8.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
390
+ "model.layers.8.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
391
+ "model.layers.8.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
392
+ "model.layers.8.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
393
+ "model.layers.8.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
394
+ "model.layers.8.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
395
+ "model.layers.8.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
396
+ "model.layers.8.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
397
+ "model.layers.8.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
398
+ "model.layers.9.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
399
+ "model.layers.9.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",
400
+ "model.layers.9.mlp.gate_proj.weight": "pytorch_model-00001-of-00003.bin",
401
+ "model.layers.9.mlp.up_proj.weight": "pytorch_model-00001-of-00003.bin",
402
+ "model.layers.9.post_attention_layernorm.weight": "pytorch_model-00001-of-00003.bin",
403
+ "model.layers.9.self_attn.k_proj.weight": "pytorch_model-00001-of-00003.bin",
404
+ "model.layers.9.self_attn.o_proj.weight": "pytorch_model-00001-of-00003.bin",
405
+ "model.layers.9.self_attn.q_proj.weight": "pytorch_model-00001-of-00003.bin",
406
+ "model.layers.9.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
407
+ "model.layers.9.self_attn.v_proj.weight": "pytorch_model-00001-of-00003.bin",
408
+ "model.norm.weight": "pytorch_model-00003-of-00003.bin"
409
+ }
410
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|startoftext|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<pad>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "clean_up_tokenization_spaces": true,
3
+ "model_max_length": 1000000000000000019884624838656,
4
+ "tokenizer_class": "PreTrainedTokenizerFast"
5
+ }