Add PPL data
Browse files
README.md
CHANGED
|
@@ -19,3 +19,17 @@ There are the following branches:
|
|
| 19 |
2_75-bpw-h6
|
| 20 |
2_55-bpw-h6
|
| 21 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
2_75-bpw-h6
|
| 20 |
2_55-bpw-h6
|
| 21 |
```
|
| 22 |
+
|
| 23 |
+
* Calibration dataset used for conversion: [wikitext-v2](https://huggingface.co/datasets/wikitext/blob/refs%2Fconvert%2Fparquet/wikitext-2-v1/test/0000.parquet)
|
| 24 |
+
* Evaluation dataset used to calculate perplexity: [wikitext-v2](https://huggingface.co/datasets/wikitext/blob/refs%2Fconvert%2Fparquet/wikitext-2-v1/validation/0000.parquet)
|
| 25 |
+
* PPL max seq. length used: 1792 (2048 with 5.0-bpw-h8 causes OOM on RTX 4090 when evaluating ppl, so had to go down a bit)
|
| 26 |
+
|
| 27 |
+
----------------------------------------------
|
| 28 |
+
| BPW | Perplexity | File Size (Gb) |
|
| 29 |
+
---------------------------------------------|
|
| 30 |
+
| 2.55-h6 | 15.0901 | 10.56 |
|
| 31 |
+
| 2.75-h6 | 13.6153 | 11.33 |
|
| 32 |
+
| 4.125-h6 | 6.8095 | 16.65 |
|
| 33 |
+
| 4.625-h6 | 6.7992 | 18.58 |
|
| 34 |
+
| 5.0-h8 | 6.7785 | 20.09 |
|
| 35 |
+
----------------------------------------------
|