---
language:
- en
pipeline_tag: text-generation
license: apache-2.0
---

# SmolLM-135M-Instruct-quantized.w4a16

## Model Overview
- **Model Architecture:** SmolLM-135M-Instruct
  - **Input:** Text
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** INT4
- **Intended Use Cases:** Intended for commercial and research use in English. Similarly to [SmolLM-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM-135M), this models is intended for assistant-like chat.
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
- **Release Date:** 8/23/2024
- **Version:** 1.0
- **License(s)**: [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
- **Model Developers:** Neural Magic

Quantized version of [SmolLM-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM-135M).
It achieves an average score of 31.91 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 31.55.

### Model Optimizations

This model was obtained by quantizing the weights of [SmolLM-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM-135M) to INT4 data type.
This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.

Only the weights of the linear operators within transformers blocks are quantized. Symmetric group-wise quantization is applied, in which a linear scaling per group maps the INT4 and floating point representations of the quantized weights.
The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library. Quantization is performed with 10% damping factor, group-size as 64 and 512 sequences sampled from [LLM Compression Calibration](https://huggingface.co/datasets/neuralmagic/LLM_compression_calibration).

## Creation

This model was created by using the [llm-compressor](https://github.com/vllm-project/llm-compressor) library as presented in the code snipet below.

```python
from transformers import AutoTokenizer
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from compressed_tensors.quantization import QuantizationArgs, QuantizationType, QuantizationStrategy
from datasets import load_dataset
import random

model_id = "HuggingFaceTB/SmolLM-135M-Instruct"


num_samples = 512
max_seq_len = 4096

tokenizer = AutoTokenizer.from_pretrained(model_id)

preprocess_fn = lambda example: {"text": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n{text}".format_map(example)}

dataset_name = "neuralmagic/LLM_compression_calibration"
dataset = load_dataset(dataset_name, split="train")
ds = dataset.shuffle().select(range(num_samples))
ds = ds.map(preprocess_fn)

examples = [
    tokenizer(
        example["text"], padding=False, max_length=max_seq_len, truncation=True,
    ) for example in ds
]

# recipe = "w4a16_nohead_recipe.yaml"
recipe = GPTQModifier(
  targets="Linear",
  scheme="W4A16",
  ignore=["lm_head"],
  dampening_frac=0.1,
)


model = SparseAutoModelForCausalLM.from_pretrained(
  model_id,
  device_map="auto",
  trust_remote_code=True
)

print(model)

oneshot(
  model=model,
  dataset=ds,
  recipe=recipe,
  max_seq_length=max_seq_len,
  num_calibration_samples=num_samples,
  oneshot_device="cuda:1,2,3",
)

model_name = model_id.split("/")[-1]

model.save_pretrained(f"{model_name}-quantized.w4a16")
tokenizer.save_pretrained(f"{model_name}-quantized.w4a16")

```


## Evaluation

The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/383bbd54bc621086e05aa1b030d8d4d5635b25e6) (commit 383bbd54bc621086e05aa1b030d8d4d5635b25e6) and the [sparseML](https://github.com/neuralmagic/sparseml) engine, using the following command:
```
lm_eval \
  --model sparseml \
  --model_args pretrained=nm-testing/SmolLM-1.7B-Instruct-quantized.w4a16,dtype=bfloat16,max_legth=2048,add_bos_token=True,parallelize=True \
  --tasks openllm \
  --batch_size auto
```

### Accuracy

#### Open LLM Leaderboard evaluation scores
<table>
  <tr>
   <td><strong>Benchmark</strong>
   </td>
   <td><strong>SmolLM-135M-Instruct </strong>
   </td>
   <td><strong>SmolLM-135M-Instruct-quantized.w4a16(this model)</strong>
   </td>
   <td><strong>Recovery</strong>
   </td>
  </tr>
  <tr>
   <td>MMLU (5-shot)
   </td>
   <td>26.220
   </td>
   <td>25.202
   </td>
   <td>96.12%
   </td>
  </tr>
  <tr>
   <td>ARC Challenge (25-shot)
   </td>
   <td>29.948
   </td>
   <td>30.034
   </td>
   <td>100.29%
   </td>
  </tr>
  <tr>
   <td>GSM-8K (5-shot, strict-match)
   </td>
   <td>1.289
   </td>
   <td>1.971
   </td>
   <td>152.91%
   </td>
  </tr>
  <tr>
   <td>Hellaswag (10-shot)
   </td>
   <td>41.41
   </td>
   <td>40.81
   </td>
   <td>98.55%
   </td>
  </tr>
  <tr>
   <td>Winogrande (5-shot)
   </td>
   <td>50.039
   </td>
   <td>53.591
   </td>
   <td>107.10%
   </td>
  </tr>
  <tr>
   <td>TruthfulQA (0-shot)
   </td>
   <td>40.38
   </td>
   <td>39.87
   </td>
   <td>98.74%
   </td>
  </tr>
  <tr>
   <td><strong>Average</strong>
   </td>
   <td><strong>31.55</strong>
   </td>
   <td><strong>31.91</strong>
   </td>
   <td><strong>101.16%</strong>
   </td>
  </tr>
</table>