shibing624
commited on
Commit
•
96507bc
1
Parent(s):
cb396cb
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,104 @@
|
|
1 |
---
|
2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
language:
|
3 |
+
- zh
|
4 |
+
tags:
|
5 |
+
- gpt2
|
6 |
+
- pytorch
|
7 |
+
- zh
|
8 |
+
- Text2Text-Generation
|
9 |
+
license: "apache-2.0"
|
10 |
+
widget:
|
11 |
+
- text: "今天你的病好点了吗?"
|
12 |
+
|
13 |
---
|
14 |
+
|
15 |
+
# GPT2 for Chinese Dialog Model(gpt2-dialogbot-base-chinese)
|
16 |
+
GPT2中文对话生成模型
|
17 |
+
|
18 |
+
`gpt2-dialogbot-base-chinese` evaluate couplet test data:
|
19 |
+
|
20 |
+
The overall performance of GPT2 on dialog **test**:
|
21 |
+
|
22 |
+
|input_text|target_text|
|
23 |
+
|:-- |:--- |
|
24 |
+
|今天你的病好点了吗?|一天比一天严重|
|
25 |
+
|
26 |
+
在常见中文闲聊数据集(包含小黄鸡语料、豆瓣语料、电视剧对白语料、贴吧论坛回帖语料、微博语料、PTT八卦语料、青云语料等)上,基本能语义正确的回答问题。
|
27 |
+
|
28 |
+
GPT2对话模型的网络结构(原生GPT2):
|
29 |
+
|
30 |
+
![arch](gpt2-model.png)
|
31 |
+
|
32 |
+
## Usage
|
33 |
+
|
34 |
+
本项目开源在对话文本生成项目:[dialogbot](https://github.com/shibing624/dialogbot),可支持GPT2模型,通过如下命令调用:
|
35 |
+
|
36 |
+
Install package:
|
37 |
+
```shell
|
38 |
+
pip install -U dialogbot
|
39 |
+
```
|
40 |
+
|
41 |
+
```shell
|
42 |
+
>>> from dialogbot import GPTBot
|
43 |
+
>>> model = GPTBot("shibing624/gpt2-dialogbot-base-chinese")
|
44 |
+
>>> r = model.answer("今天你的病好点了吗?")
|
45 |
+
'一天比一天严重'
|
46 |
+
```
|
47 |
+
|
48 |
+
|
49 |
+
模型文件组成:
|
50 |
+
```
|
51 |
+
gpt2-dialogbot-base-chinese
|
52 |
+
├── config.json
|
53 |
+
├── pytorch_model.bin
|
54 |
+
└── vocab.txt
|
55 |
+
```
|
56 |
+
|
57 |
+
|
58 |
+
### 训练数据集
|
59 |
+
#### 中文对话数据集
|
60 |
+
|
61 |
+
### 闲聊语料分享
|
62 |
+
|中文闲聊语料 | 数据集地址 |语料描述|
|
63 |
+
|---------|--------|--------|
|
64 |
+
|常见中文闲聊|[chinese_chatbot_corpus](https://github.com/codemayq/chinese_chatbot_corpus)|包含小黄鸡语料、豆瓣语料、电视剧对白语料、贴吧论坛回帖语料、微博语料、PTT八卦语料、青云语料等|
|
65 |
+
|50w中文闲聊语料 | [百度网盘【提取码:4g5e】](https://pan.baidu.com/s/1M87Zf9e8iBqqmfTkKBWBWA) 或 [GoogleDrive](https://drive.google.com/drive/folders/1QFRsftLNTR_D3T55mS_FocPEZI7khdST?usp=sharing) |包含50w个多轮对话的原始语料、预处理数据|
|
66 |
+
|100w中文闲聊语料 | [百度网盘【提取码:s908】](https://pan.baidu.com/s/1TvCQgJWuOoK2f5D95nH3xg) 或 [GoogleDrive](https://drive.google.com/drive/folders/1NU4KLDRxdOGINwxoHGWfVOfP0wL05gyj?usp=sharing)|包含100w个多轮对话的原始语料、预处理数据|
|
67 |
+
|
68 |
+
|
69 |
+
|
70 |
+
- [dialogbot](https://github.com/shibing624/dialogbot)
|
71 |
+
|
72 |
+
|
73 |
+
数据格式:
|
74 |
+
|
75 |
+
```text
|
76 |
+
谢谢你所做的一切
|
77 |
+
你开心就好
|
78 |
+
开心
|
79 |
+
嗯因为你的心里只有学习
|
80 |
+
某某某,还有你
|
81 |
+
这个某某某用的好
|
82 |
+
|
83 |
+
你们宿舍都是这么厉害的人吗
|
84 |
+
眼睛特别搞笑这土也不好捏但就是觉得挺可爱
|
85 |
+
特别可爱啊
|
86 |
+
```
|
87 |
+
|
88 |
+
|
89 |
+
如果需要训练GPT2模型,请参考[https://github.com/shibing624/dialogbot](https://github.com/shibing624/dialogbot)
|
90 |
+
|
91 |
+
|
92 |
+
## Citation
|
93 |
+
|
94 |
+
如果你在研究中使用了dialogbot,请按如下格式引用:
|
95 |
+
|
96 |
+
```latex
|
97 |
+
@software{dialogbot,
|
98 |
+
author = {Xu Ming},
|
99 |
+
title = {dialogbot: Dialogue Model Technology Tool},
|
100 |
+
year = {2021},
|
101 |
+
url = {https://github.com/shibing624/dialogbot},
|
102 |
+
}
|
103 |
+
```
|
104 |
+
|