Hugging Face's logo

baichuan-inc
/
Baichuan-13B-Base

Text Generation
Transformers
PyTorch
Chinese
English
baichuan
custom_code
text-generation-inference
Model card Files Community
3
s-JoL commited on
Commit
51aa849
•
1 Parent(s): ee077eb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +60 -112
README.md CHANGED
@@ -5,28 +5,19 @@ language:
5
  pipeline_tag: text-generation
6
  inference: false
7
  ---
8
- # baichuan-7B
9
 
10
  <!-- Provide a quick summary of what the model is/does. -->
11
 
12
- baichuan-7B是由百川智能开发的一个开源的大规模预训练模型。基于Transformer结构，在大约1.2万亿tokens上训练的70亿参数模型，支持中英双语，上下文窗口长度为4096。在标准的中文和英文权威benchmark（C-EVAL/MMLU）上均取得同尺寸最好的效果。
 
13
 
14
- 如果希望使用baichuan-7B（如进行推理、Finetune等），我们推荐使用配套代码库[baichuan-7B](https://
15
- .com/baichuan-inc/baichuan-7B)。
 
 
16
 
17
- baichuan-7B is an open-source large-scale pre-trained model developed by Baichuan Intelligent Technology. Based on the Transformer architecture, it is a model with 7 billion parameters trained on approximately 1.2 trillion tokens. It supports both Chinese and English, with a context window length of 4096. It achieves the best performance of its size on standard Chinese and English authoritative benchmarks (C-EVAL/MMLU).
18
 
19
- If you wish to use baichuan-7B (for inference, finetuning, etc.), we recommend using the accompanying code library [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B).
20
-
21
- ## Why use baichuan-7B
22
-
23
- - 在同尺寸模型中baichuan-7B达到了目前SOTA的水平，参考下面MMLU指标
24
- - baichuan-7B使用自有的中英文双语语料进行训练，在中文上进行优化，在C-Eval达到SOTA水平
25
- - 不同于LLaMA完全禁止商业使用，baichuan-7B使用更宽松的开源协议，允许用于商业目的
26
-
27
- - Among models of the same size, baichuan-7B has achieved the current state-of-the-art (SOTA) level, as evidenced by the following MMLU metrics.
28
- - baichuan-7B is trained on proprietary bilingual Chinese-English corpora, optimized for Chinese, and achieves SOTA performance on C-Eval.
29
- - Unlike LLaMA, which completely prohibits commercial use, baichuan-7B employs a more lenient open-source license, allowing for commercial purposes.
30
 
31
  ## How to Get Started with the Model
32
 
@@ -63,7 +54,7 @@ print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
63
  - **Developed by:** 百川智能(Baichuan Intelligent Technology)
64
  - **Email**: [email protected]
65
  - **Language(s) (NLP):** Chinese/English
66
- - **License:** [baichuan-7B License](https://huggingface.co/baichuan-inc/baichuan-7B/blob/main/baichuan-7B%20%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE.pdf)
67
 
68
  ### Model Sources
69
 
@@ -107,7 +98,7 @@ The specific parameters are as follows:
107
  ### Downstream Use
108
 
109
  <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
110
- 我们同时开源出了和本模型配套的训练代码，允许进行高效的Finetune用于下游任务，具体参见[baichuan-7B](https://github.com/baichuan-inc/baichuan-7B)。
111
 
112
  We have also open-sourced the training code that accompanies this model, allowing for efficient finetuning for downstream tasks. For more details, please refer to [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B).
113
 
@@ -122,108 +113,65 @@ Production use without adequate assessment of risks and mitigation; any use case
122
 
123
  <!-- This section is meant to convey both technical and sociotechnical limitations. -->
124
 
125
- baichuan-7B可能会产生事实上不正确的输出，不应依赖它产生事实上准确的信息。baichuan-7B是在各种公共数据集上进行训练的。尽管我们已经做出了巨大的努力来清洗预训练数据，但这个模型可能会生成淫秽、偏见或其他冒犯性的输出。
126
 
127
- baichuan-7B can produce factually incorrect output, and should not be relied on to produce factually accurate information. baichuan-7B was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
128
 
129
  ## Training Details
130
 
131
- 训练具体设置参见[baichuan-7B](https://github.com/baichuan-inc/baichuan-7B)。
132
 
133
- For specific training settings, please refer to [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B).
134
 
135
  ## Evaluation
136
 
137
- ### 中文评测
138
- #### C-Eval
139
- [CEval数据集](https://cevalbenchmark.com/index.html)是一个全面的中文基础模型评测数据集，涵盖了52个学科和四个难度的级别。我们使用该数据集的dev集作为few-shot的来源，在test集上进行了5-shot测试。
140
-
141
-
142
- | Model 5-shot | Average | Avg(Hard) | STEM | Social Sciences | Humanities | Others |
143
- |-----------------------------|---------|-----------|------|-----------------|------------|--------|
144
- | GPT-4 | 68.7 | 54.9 | 67.1 | 77.6 | 64.5 | 67.8 |
145
- | ChatGPT | 54.4 | 41.4 | 52.9 | 61.8 | 50.9 | 53.6 |
146
- | Claude-v1.3 | 54.2 | 39.0 | 51.9 | 61.7 | 52.1 | 53.7 |
147
- | Claude-instant-v1.0 | 45.9 | 35.5 | 43.1 | 53.8 | 44.2 | 45.4 |
148
- | moss-moon-003-base (16B) | 27.4 | 24.5 | 27.0 | 29.1 | 27.2 | 26.9 |
149
- | Ziya-LLaMA-13B-pretrain | 30.2 | 22.7 | 27.7 | 34.4 | 32.0 | 28.9 |
150
- | LLaMA-7B-hf | 27.1 | 25.9 | 27.1 | 26.8 | 27.9 | 26.3 |
151
- | ChatGLM-6B | 34.5 | 23.1 | 30.4 | 39.6 | 37.4 | 34.5 |
152
- | Falcon-7B | 25.8 | 24.3 | 25.8 | 26.0 | 25.8 | 25.6 |
153
- | Open-LLaMA-v2-pretrain (7B) | 24.0 | 22.5 | 23.1 | 25.3 | 25.2 | 23.2 |
154
- | TigerBot-7B-base | 25.7 | 27.0 | 27.3 | 24.7 | 23.4 | 26.1 |
155
- | Aquila-7B<sup>*</sup> | 25.5 | 25.2 | 25.6 | 24.6 | 25.2 | 26.6 |
156
- | BLOOM-7B | 22.8 | 20.2 | 21.8 | 23.3 | 23.9 | 23.3 |
157
- | BLOOMZ-7B | 35.7 | 25.8 | 31.3 | 43.5 | 36.6 | 35.6 |
158
- | **baichuan-7B** | 42.8 | 31.5 | 38.2 | 52.0 | 46.2 | 39.3 |
159
-
160
-
161
- #### Gaokao
162
- [Gaokao](https://github.com/ExpressAI/AI-Gaokao) 是一个以中国高考题作为评测大语言模型能力的数据集，用以评估模型的语言能力和逻辑推理能力。
163
- 我们只保留了其中的单项选择题，并对所有模型进行统一5-shot测试。
164
-
165
- 以下是测试的结果。
166
-
167
- | Model | Average |
168
- |-------------------------|-----------------|
169
- | Open-LLaMA-v2-pretrain | 21.41 |
170
- | Ziya-LLaMA-13B-pretrain | 23.17 |
171
- | Falcon-7B | 23.98 |
172
- | TigerBot-7B-base | 25.94 |
173
- | LLaMA-7B | 27.81 |
174
- | ChatGLM-6B | 21.41 |
175
- | BLOOM-7B | 26.96 |
176
- | BLOOMZ-7B | 28.72 |
177
- | Aquila-7B<sup>*</sup> | 24.39 |
178
- | **baichuan-7B** | **36.24** |
179
-
180
-
181
- #### AGIEval
182
- [AGIEval](https://github.com/microsoft/AGIEval) 旨在评估模型的认知和解决问题相关的任务中的一般能力。
183
- 我们只保留了其中的四选一单项选择题，随机划分后对所有模型进行了统一5-shot测试。
184
-
185
- | Model | Average |
186
- |-------------------------|-----------------|
187
- | Open-LLaMA-v2-pretrain | 23.49 |
188
- | Ziya-LLaMA-13B-pretrain | 27.64 |
189
- | Falcon-7B | 27.18 |
190
- | TigerBot-7B-base | 25.19 |
191
- | LLaMA-7B | 28.17 |
192
- | ChatGLM-6B | 23.49 |
193
- | BLOOM-7B | 26.55 |
194
- | BLOOMZ-7B | 30.27 |
195
- | Aquila-7B<sup>*</sup> | 25.58 |
196
- | **baichuan-7B** | **34.44** |
197
-
198
- <sup>*</sup>其中Aquila模型来源于[智源官方网站](https://model.baai.ac.cn/model-detail/100098)，仅做参考
199
-
200
- ### English Leaderboard
201
- In addition to Chinese, we also tested the model's performance in English.
202
-
203
- #### MMLU
204
-
205
- [MMLU](https://arxiv.org/abs/2009.03300) is an English evaluation dataset that includes 57 multiple-choice tasks, covering elementary mathematics, American history, computer science, law, etc. The difficulty ranges from high school level to expert level, making it a mainstream LLM evaluation dataset.
206
-
207
- We adopted the [open-source]((https://github.com/hendrycks/test)) evaluation scheme, and the final 5-shot results are as follows:
208
-
209
- | Model | Humanities | Social Sciences | STEM | Other | Average |
210
- |----------------------------------------|-----------:|:---------------:|:----:|:-----:|:-------:|
211
- | LLaMA-7B<sup>2</sup> | 34.0 | 38.3 | 30.5 | 38.1 | 35.1 |
212
- | Falcon-7B<sup>1</sup> | - | - | - | - | 35.0 |
213
- | mpt-7B<sup>1</sup> | - | - | - | - | 35.6 |
214
- | ChatGLM-6B<sup>0</sup> | 35.4 | 41.0 | 31.3 | 40.5 | 36.9 |
215
- | BLOOM 7B<sup>0</sup> | 25.0 | 24.4 | 26.5 | 26.4 | 25.5 |
216
- | BLOOMZ 7B<sup>0</sup> | 31.3 | 42.1 | 34.4 | 39.0 | 36.1 |
217
- | moss-moon-003-base (16B)<sup>0</sup> | 24.2 | 22.8 | 22.4 | 24.4 | 23.6 |
218
- | moss-moon-003-sft (16B)<sup>0</sup> | 30.5 | 33.8 | 29.3 | 34.4 | 31.9 |
219
- | **baichuan-7B<sup>0</sup>** | 38.4 | 48.9 | 35.6 | 48.1 | 42.3 |
220
-
221
- The superscript in the Model column indicates the source of the results.
222
- ```
223
- 0:reimplemented
224
- 1:https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
225
- 2:https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu
226
- ```
227
 
228
  ## Our Group
229
  [WeChat](https://github.com/baichuan-inc/baichuan-7B/blob/main/media/wechat.jpeg?raw=true)
 
5
  pipeline_tag: text-generation
6
  inference: false
7
  ---
8
+ # Baichuan-13B
9
 
10
  <!-- Provide a quick summary of what the model is/does. -->
11
 
12
+ ## 介绍
13
+ Baichuan-13B 是由百川智能继 [Baichuan-7B](https://github.com/baichuan-inc/baichuan-7B) 之后开发的包含 130 亿参数的开源可商用的大规模语言模型，在标准的中文和英文 benchmark上均取得同尺寸最好的效果。本次发布包含有预训练 (Baichuan-13B-Base) 和对齐 (Baichuan-13B-Chat) 两个版本。Baichuan-13B 有如下几个特点：
14
 
15
+ 1. **开源可商用百亿级别中文语言模型**：Baichuan-13B-Base 是免费开源可商用的百亿级别中文预训练语言模型。包含有130亿参数，没有经过任何 Instruction Tuning 或者针对 benchmark 的优化，纯净、高可定制。弥补了在中文领域缺乏 100 亿以上高可用中文预训练大模型的短板。
16
+ 2. **更大尺寸、更多数据**：在 Baichuan-7B 的基础上进一步扩大参数量到 130 亿，并且在高质量的语料上训练了 1.4 万亿 tokens，是当前开源 13B 尺寸下训练数据量最多的模型。支持中英双语，使用 [ALiBi](https://arxiv.org/abs/2108.12409) 位置编码，上下文窗口长度为 4096。
17
+ 3. **同时开源预训练和对齐模型**：预训练模型是适用开发者的”基座“，而广大普通用户对有对话功能的对齐模型具有更强的需求。因此本次开源我们同时发布了对齐模型（Baichuan-13B-Chat），具有很强的对话能力，开箱即用，支持很简单的部署。
18
+ 4. **更高效的推理**：为了支持更广大用户的使用，我们本次同时开源了 int8 和 int4 的量化版本，在几乎没有效果损失的情况下可以很方便的将模型部署在低显存机器上。
19
 
 
20
 
 
 
 
 
 
 
 
 
 
 
 
21
 
22
  ## How to Get Started with the Model
23
 
 
54
  - **Developed by:** 百川智能(Baichuan Intelligent Technology)
55
  - **Email**: [email protected]
56
  - **Language(s) (NLP):** Chinese/English
57
+ - **License:** [Baichuan-13B License]()
58
 
59
  ### Model Sources
60
 
 
98
  ### Downstream Use
99
 
100
  <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
101
+ 我们同时开源出了和本模型配套的训练代码，允许进行高效的Finetune用于下游任务，具体参见[Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B)。
102
 
103
  We have also open-sourced the training code that accompanies this model, allowing for efficient finetuning for downstream tasks. For more details, please refer to [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B).
104
 
 
113
 
114
  <!-- This section is meant to convey both technical and sociotechnical limitations. -->
115
 
116
+ Baichuan-7B可能会产生事实上不正确的输出，不应依赖它产生事实上准确的信息。Baichuan-7B是在各种公共数据集上进行训练的。尽管我们已经做出了巨大的努力来清洗预训练数据，但这个模型可能会生成淫秽、偏见或其他冒犯性的输出。
117
 
118
+ Baichuan-7B can produce factually incorrect output, and should not be relied on to produce factually accurate information. Baichuan-7B was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
119
 
120
  ## Training Details
121
 
122
+ 训练具体设置参见[Baichuan-7B](https://github.com/baichuan-inc/Baichuan-7B)。
123
 
124
+ For specific training settings, please refer to [Baichuan-7B](https://github.com/baichuan-inc/Baichuan-7B).
125
 
126
  ## Evaluation
127
 
128
+ # Benchmark结果
129
+
130
+ 我们在各个 benchmark 下进行了`5-shot`评测，所采用的方法和 [Baichuan-7B](https://github.com/baichuan-inc/Baichuan-7B/) 项目中相同。结果如下：
131
+
132
+ ## C-Eval
133
+
134
+ | Model 5-shot | STEM | Social Sciences | Humanities | Others | Average |
135
+ |-------------------------|-------|-----------------|------------|--------|---------|
136
+ | ChatGLM2-6B | 45.9 | 61.6 | 49.7 | 48.2 | 50.2 |
137
+ | InternLM-7B<sup>*</sup> | 40.1 | 55.7 | 49.4 | 37.9 | 44.6 |
138
+ | Baichuan-7B | 38.2 | 52.0 | 46.2 | 39.3 | 42.8 |
139
+ | Ziya-LLaMA-13B-Pretrain | 27.6 | 34.4 | 32.0 | 28.6 | 30.0 |
140
+ | LLaMA-13B | 27.0 | 33.6 | 27.7 | 27.6 | 28.5 |
141
+ | moss-moon-003-base (16B)| 27.0 | 29.1 | 27.2 | 26.9 | 27.4 |
142
+ | vicuna-13B | 22.8 | 24.8 | 22.3 | 18.5 | 22.2 |
143
+ | **Baichuan-13B-Base** | **45.9** | **63.5** | **57.2** | **49.3** | **52.4** |
144
+ | **Baichuan-13B-Chat** | **43.7** | **64.6** | **56.2** | **49.2** | **51.5** |
145
+ > *说明：表中各个模型的结果是使用统一的评估代码得到。[InternLM-7B](https://huggingface.co/internlm/internlm-7b) 汇报使用 [OpenCompass](https://opencompass.org.cn/rank) 工具评估的C-Eval平均值为 53.4，我们使用 OpenCompass 评估 InternLM-7B 的平均值为 51.6
146
+
147
+ ## MMLU
148
+
149
+ | Model 5-shot | STEM | Social Sciences | Humanities | Others | Average |
150
+ |-------------------------|-------|-----------------|------------|--------|---------|
151
+ | LLaMA-13B | 36.1 | 53.0 | 44.0 | 52.8 | 46.3 |
152
+ | ChatGLM2-6B | 38.2 | 52.5 | 43.2 | 50.8 | 45.9 |
153
+ | InternLM-7B | 38.0 | 51.1 | 39.2 | 50.2 | 44.1 |
154
+ | Ziya-LLaMA-13B-Pretrain | 35.6 | 47.6 | 40.1 | 49.4 | 42.9 |
155
+ | Baichuan-7B | 35.6 | 48.9 | 38.4 | 48.1 | 42.3 |
156
+ | vicuna-13B | 24.2 | 24.1 | 24.6 | 26.8 | 24.9 |
157
+ | moss-moon-003-base (16B)| 22.4 | 22.8 | 24.2 | 24.4 | 23.6 |
158
+ | **Baichuan-13B-Base** | **41.6** | **60.9** | **47.4** | **58.5** | **51.6** |
159
+ | **Baichuan-13B-Chat** | **40.9** | **60.9** | **48.8** | **59.0** | **52.1** |
160
+
161
+
162
+ ## CMMLU
163
+
164
+ | Model 5-shot | STEM | Humanities | Social Sciences | Others | China Specific | Average |
165
+ |-------------------------|-------|------------|-----------------|--------|----------------|---------|
166
+ | InternLM-7B | 41.7 | 54.4 | 56.4 | 55.4 | 53.1 | 52.1 |
167
+ | ChatGLM2-6B | 42.5 | 51.4 | 51.4 | 50.7 | 48.4 | 49.0 |
168
+ | Baichuan-7B | 34.4 | 47.5 | 47.6 | 46.6 | 44.3 | 44.0 |
169
+ | Ziya-LLaMA-13B-Pretrain | 29.0 | 30.7 | 33.8 | 34.4 | 31.9 | 32.1 |
170
+ | LLaMA-13B | 29.2 | 30.8 | 31.6 | 33.0 | 30.5 | 31.2 |
171
+ | moss-moon-003-base (16B)| 27.2 | 30.4 | 28.8 | 32.6 | 28.7 | 29.6 |
172
+ | vicuna-13B | 24.0 | 25.4 | 25.3 | 25.0 | 25.0 | 24.9 |
173
+ | **Baichuan-13B-Base** | **41.7** | **61.1** | **59.8** | **59.0** | **56.4** | **55.3** |
174
+ | **Baichuan-13B-Chat** | **42.8** | **62.6** | **59.7** | **59.0** | **56.1** | **55.8** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
175
 
176
  ## Our Group
177
  [WeChat](https://github.com/baichuan-inc/baichuan-7B/blob/main/media/wechat.jpeg?raw=true)