Hugging Face's logo

Dracones
/
miqu-1-70b-sf_exl2_2.25bpw

Text Generation
Transformers
Safetensors
English
llama
exl2
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
miqu-1-70b-sf_exl2_2.25bpw
File size: 4,020 Bytes 
8c99c80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ee1f05a
 
 
 
 
 
 
 
 
 
8c99c80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
 
---
library_name: transformers
language:
- en
license: unknown
tags:
- exl2
---

# miqu-1-70b-sf - EXL2 2.25bpw

This is a 2.25bpw EXL2 quant of [152334H/miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf)

Details about the model can be found at the above model page.


## EXL2 Version

These quants were made with exllamav2 version 0.0.18. Quants made on this version of EXL2 may not work on older versions of the exllamav2 library.

If you have problems loading these models, please update Text Generation WebUI to the latest version.

## Perplexity Scoring

Below are the perplexity scores for the EXL2 models. A lower score is better. 
 
| Quant Level | Perplexity Score |
|-------------|------------------|
| 5.0 | 4.2637 |
| 4.5 | 4.2876 |
| 4.0 | 4.3097 |
| 3.5 | 4.4459 |
| 3.0 | 4.6504 |
| 2.75 | 5.1638 |
| 2.5 | 5.1715 |
| 2.25 | 6.0848 |

## EQ Bench

Here are the EQ Bench scores for the EXL2 quants using Alpaca, ChatML, Mistral, Vicuna-v1.1 and Vicuna-v0 prompt templates. A higher score is better.

| Quant Size | ChatML | Alpaca | Mistral | Vicuna-v1.1 | Vicuna-v0 |
|------------|--------|--------|---------|-------------|-----------|
| 5.0        | 79.91  | 81.45  | 81.11   | 78.37       | 76.64     |
| 4.5        | 80.64  | 80.9   | 81.65   | 77.04       | 74.6      |
| 4.0        | 80.78  | 79.53  | 82.78   | 79.17       | 76.41     |
| 3.5        | 81.11  | 82.42  | 82.34   | 81.04       | 78.09     |
| 3.0        | 79.13  | 77.74  | 80.11   | 79.38       | 77.25     |
| 2.75       | 79.6   | 77.85  | 79.71   | 76.93       | 75.91     |
| 2.5        | 77.45  | 77.0   | 78.4    | 75.86       | 75.25     |
| 2.25       | 77.18  | 74.06  | 76.75   | 75.56       | 74.28     |


### Perplexity Script

This was the script used for perplexity testing.

```bash
#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Set the model name and bit size
MODEL_NAME="miqu-1-70b-sf"
BIT_PRECISIONS=(8.0 7.0 6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.75 2.5 2.25)

# Print the markdown table header
echo "| Quant Level | Perplexity Score |"
echo "|-------------|------------------|"

for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
  MODEL_DIR="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"
  if [ -d "$MODEL_DIR" ]; then
    output=$(python test_inference.py -m "$MODEL_DIR" -gs 22,24 -ed data/wikitext/wikitext-2-v1.parquet)
    score=$(echo "$output" | grep -oP 'Evaluation perplexity: \K[\d.]+')
    echo "| $BIT_PRECISION | $score |"
  fi
done
```


## Quant Details

This is the script used for quantization.

```bash
#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Set the model name and bit size
MODEL_NAME="miqu-1-70b-sf"

# Define variables
MODEL_DIR="models/152334H_miqu-1-70b-sf"
OUTPUT_DIR="exl2_$MODEL_NAME"
MEASUREMENT_FILE="measurements/$MODEL_NAME.json"

# Create the measurement file if needed
if [ ! -f "$MEASUREMENT_FILE" ]; then
    echo "Creating $MEASUREMENT_FILE"
    # Create directories
    if [ -d "$OUTPUT_DIR" ]; then
        rm -r "$OUTPUT_DIR"
    fi
    mkdir "$OUTPUT_DIR"

    python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -om $MEASUREMENT_FILE
fi

# Choose one of the below. Either create a single quant for testing or a batch of them.
# BIT_PRECISIONS=(5.0)
BIT_PRECISIONS=(8.0 7.0 6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.75 2.5 2.25)

for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
    CONVERTED_FOLDER="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"

    # If it doesn't already exist, make the quant
    if [ ! -d "$CONVERTED_FOLDER" ]; then

        echo "Creating $CONVERTED_FOLDER"

        # Create directories
        if [ -d "$OUTPUT_DIR" ]; then
            rm -r "$OUTPUT_DIR"
        fi
        mkdir "$OUTPUT_DIR"
        mkdir "$CONVERTED_FOLDER"
        
        # Run conversion commands  
        python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -m $MEASUREMENT_FILE -b $BIT_PRECISION -cf $CONVERTED_FOLDER

    fi
done
```