Update README.md

README.md

@@ -29,7 +29,7 @@ This model was obtained by quantizing the weights of [Phi-3-medium-4k-instruct](
 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 25%.
 
 Only the weights of the linear operators within transformers blocks are quantized. Symmetric group-wise quantization is applied, in which a linear scaling per group maps the INT4 and floating point representations of the quantized weights.
-[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) is used for quantization with 1% damping factor, group-size as 128 and 512 sequences sampled from [Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus).
+The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library. Quantization is performed with 1% damping factor, group-size as 128 and 512 sequences sampled from [Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus).
 
 
 ## Deployment
@@ -63,12 +63,11 @@ generated_text = outputs[0].outputs[0].text
 print(generated_text)
 ```
 
-vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
+vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
 
 ### Use with transformers
 
-This model is supported by Transformers leveraging the integration with the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) data format.
-The following example contemplates how the model can be used using the `generate()` function.
+The following example contemplates how the model can be deployed in Transformers using the `generate()` function.
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -112,12 +111,12 @@ print(tokenizer.decode(response, skip_special_tokens=True))
 
 ## Creation
 
-This model was created by applying the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) library as presented in the code snipet below.
-Although AutoGPTQ was used for this particular model, Neural Magic is transitioning to using [llm-compressor](https://github.com/vllm-project/llm-compressor) which supports several quantization schemes and models not supported by AutoGPTQ.
+This model was created by using the [llm-compressor](https://github.com/vllm-project/llm-compressor) library as presented in the code snipet below.
 
 ```python
 from transformers import AutoTokenizer
-from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
+from llmcompressor.modifiers.quantization import GPTQModifier
 from datasets import load_dataset
 import random
 
@@ -141,21 +140,27 @@ examples = [
     ) for example in ds
 ]
 
-quantize_config = BaseQuantizeConfig(
-  bits=4,
-  group_size=128,
-  desc_act=True,
-  model_file_base_name="model",
-  damp_percent=0.01,
+recipe = GPTQModifier(
+  targets="Linear",
+  scheme="W4A16",
+  ignore=["lm_head"],
+  dampening_frac=0.1,
 )
 
-model = AutoGPTQForCausalLM.from_pretrained(
+model = SparseAutoModelForCausalLM.from_pretrained(
   model_id,
-  quantize_config,
   device_map="auto",
+  trust_remote_code=True,
+)
+
+oneshot(
+  model=model,
+  dataset=ds,
+  recipe=recipe,
+  max_seq_length=max_seq_len,
+  num_calibration_samples=num_samples,
 )
 
-model.quantize(examples)
 model.save_pretrained("Phi-3-medium-128k-instruct-quantized.w4a16")
 ```