---
base_model: NousResearch/Nous-Hermes-2-Yi-34B
inference: true
model_type: yi
quantized_by: robertgshaw2
tags:
- nm-vllm
- marlin
- int4
---

## Nous-Hermes-Yi-34B-marlin
This repo contains model files for [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) optimized for [nm-vllm](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.

This model was quantized with [GPTQ](https://arxiv.org/abs/2210.17323) and saved in the Marlin format for efficient 4-bit inference. Marlin is a highly optimized inference kernel for 4 bit models.

## Inference
Install [nm-vllm](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
```bash
pip install nm-vllm[sparse]
```

Run in a Python pipeline for local inference:
```python
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_id = "neuralmagic/Nous-Hermes-2-Yi-34B-marlin"
model = LLM(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
    {"role": "user", "content": "What is synthetic data in machine learning?"},
]
formatted_prompt =  tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
sampling_params = SamplingParams(max_tokens=200)
outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

"""
Synthetic data is data that has been artificially created or modified to serve the needs of machine learning and data analysis tasks. It can be generated either through title methods like stochastic simulations or through processes of data augmentation that take original data and modify/manipulate it to create new samples. Synthetic data is often used in machine learning when the available amount of real-world data is insufficient or in cases where the creation of real-world data can be dangerous, costly, or time-consuming.
"""
```

## Quantization
For details on how this model was quantized and converted to marlin format, run the `quantization/apply_gptq_save_marlin.py` script:

```bash
pip install -r quantization/requirements.txt
python3 quantization/apply_gptq_save_marlin.py --model-id NousResearch/Nous-Hermes-2-Yi-34B --save-dir ./nous-hermes-2-yi-34b-marlin
```

## Slack

For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)