yinsong1986
commited on
Commit
•
f8e22b4
1
Parent(s):
5cc44e3
Update README.md
Browse files
README.md
CHANGED
@@ -10,7 +10,7 @@ MistralLite is a fine-tuned [Mistral-7B-v0.1](https://huggingface.co/mistralai/M
|
|
10 |
MistralLite is similar to [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), and their similarities and differences are summarized below:
|
11 |
|Model|Fine-tuned on long contexts| Max context length| RotaryEmbedding adaptation| Sliding Window Size|
|
12 |
|----------|-------------:|------------:|-----------:|-----------:|
|
13 |
-
| Mistral-7B-v0.1 | up to 8K tokens | 32K | rope_theta = 10000 | 4096 |
|
14 |
| MistralLite | up to 16K tokens | 32K | **rope_theta = 1000000** | **16384** |
|
15 |
|
16 |
## Motivation of Developing MistralLite
|
|
|
10 |
MistralLite is similar to [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), and their similarities and differences are summarized below:
|
11 |
|Model|Fine-tuned on long contexts| Max context length| RotaryEmbedding adaptation| Sliding Window Size|
|
12 |
|----------|-------------:|------------:|-----------:|-----------:|
|
13 |
+
| Mistral-7B-Instruct-v0.1 | up to 8K tokens | 32K | rope_theta = 10000 | 4096 |
|
14 |
| MistralLite | up to 16K tokens | 32K | **rope_theta = 1000000** | **16384** |
|
15 |
|
16 |
## Motivation of Developing MistralLite
|