mikecovlee
commited on
Commit
•
d41e574
1
Parent(s):
bc88d8b
Update README.md
Browse files
README.md
CHANGED
@@ -2,4 +2,84 @@
|
|
2 |
license: apache-2.0
|
3 |
datasets:
|
4 |
- yahma/alpaca-cleaned
|
5 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2 |
license: apache-2.0
|
3 |
datasets:
|
4 |
- yahma/alpaca-cleaned
|
5 |
+
---
|
6 |
+
# MixLoRA: Resource-Efficient Model with Mix-of-Experts Architecture for Enhanced LoRA Performance
|
7 |
+
|
8 |
+
<div align="left"><img src="./Mix-LoRA.png" width=60%"></div>
|
9 |
+
|
10 |
+
GitHub: https://github.com/TUDB-Labs/multi-lora-fine-tune
|
11 |
+
|
12 |
+
The fundamental concept of MixLoRA is based on a pre-trained model with all parameters frozen, such as LLaMA-7B. It involves training multiple LoRA expert modules on top of its fully connected layer (FFN). Simultaneously, a routing layer (Gate Linear) is trained, creating a more powerful Mixture of Experts (MoE) language model. Theoretically, this approach allows achieving performance similar to existing MoE models but with fewer resources.
|
13 |
+
|
14 |
+
In addition, MixLoRA also allows simultaneous fine-tuning of the attention layer, contributing to improved fine-tuning outcomes. In experiments, when the attention layer is fine-tuned simultaneously, the MixLoRA model composed of 8 experts exhibits a faster rate of loss reduction compared to the MixLoRA model with 9 experts.
|
15 |
+
|
16 |
+
MixLoRA exists within m-LoRA in a specific adapter form. Consequently, m-LoRA is capable of simultaneously loading, training, and fine-tuning multiple distinct MixLoRA models. However, it's essential to note that these models must be based on the same pre-trained model.
|
17 |
+
|
18 |
+
## Create MixLoRA model
|
19 |
+
|
20 |
+
Basic command for creating a baseline model on the [Alpaca Cleaned](https://github.com/gururise/AlpacaDataCleaned) dataset:
|
21 |
+
```bash
|
22 |
+
python mlora.py \
|
23 |
+
--base_model yahma/llama-7b-hf \
|
24 |
+
--config ./config/alpaca_mixlora.json \
|
25 |
+
--load_8bit \
|
26 |
+
--mixlora
|
27 |
+
```
|
28 |
+
Please note that once the MixLoRA model is created, the number of experts in the model cannot be changed.
|
29 |
+
|
30 |
+
## Fine-tuning MixLoRA model
|
31 |
+
|
32 |
+
The MixLoRA model can also undergo further fine-tuning.
|
33 |
+
Basic command for finetuning a model on the [Alpaca Cleaned](https://github.com/gururise/AlpacaDataCleaned) dataset:
|
34 |
+
```bash
|
35 |
+
python mlora.py \
|
36 |
+
--base_model yahma/llama-7b-hf \
|
37 |
+
--config ./config/alpaca_mixlora.json \
|
38 |
+
--load_adapter \
|
39 |
+
--load_8bit \
|
40 |
+
--mixlora
|
41 |
+
```
|
42 |
+
|
43 |
+
## Evaluate MixLoRA model
|
44 |
+
|
45 |
+
Currently, MixLoRA supports evaluation only through the m-LoRA framework.
|
46 |
+
```bash
|
47 |
+
python mlora.py \
|
48 |
+
--base_model yahma/llama-7b-hf \
|
49 |
+
--config ./config/alpaca_mixlora.json \
|
50 |
+
--load_adapter \
|
51 |
+
--load_8bit \
|
52 |
+
--inference \
|
53 |
+
--mixlora
|
54 |
+
```
|
55 |
+
|
56 |
+
## Citation
|
57 |
+
Please cite the repo if you use the code in this repo.
|
58 |
+
```bibtex
|
59 |
+
@misc{Mix-LoRA,
|
60 |
+
author = {Dengchun, Li and Tingfeng, Lan and Zhengmao, Ye and Lei, Duan and Mingjie, Tang},
|
61 |
+
title = {MixLoRA: Resource-Efficient Model with Mix-of-Experts Architecture for Enhanced LoRA Performance},
|
62 |
+
year = {2023},
|
63 |
+
publisher = {GitHub},
|
64 |
+
howpublished = {\url{https://github.com/TUDB-Labs/multi-lora-fine-tune}},
|
65 |
+
}
|
66 |
+
```
|
67 |
+
|
68 |
+
## Copyright
|
69 |
+
Copyright © 2023 All Rights Reserved.
|
70 |
+
|
71 |
+
This project is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
|
72 |
+
|
73 |
+
```
|
74 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
75 |
+
you may not use this file except in compliance with the License.
|
76 |
+
You may obtain a copy of the License at
|
77 |
+
|
78 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
79 |
+
|
80 |
+
Unless required by applicable law or agreed to in writing, software
|
81 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
82 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
83 |
+
See the License for the specific language governing permissions and
|
84 |
+
limitations under the License.
|
85 |
+
```
|