|
--- |
|
license: llama3.1 |
|
language: |
|
- en |
|
base_model: |
|
- meta-llama/Meta-Llama-3.1-8B-Instruct |
|
--- |
|
# Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit |
|
- Model creator: [Meta-Llama](https://huggingface.co/meta-llama) |
|
- Original model: [Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) |
|
|
|
|
|
## Overview |
|
This repository contains a 4-bit AWQ version of **Meta-Llama-3.1-8B-Instruct**, optimized for the LMDeploy TurboMindEngine. |
|
The model is designed to provide efficient and accurate performance with reduced computational requirements. |
|
|
|
## Model Details |
|
- **Model Name**: Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit |
|
- **Base Model**: meta-llama/Meta-Llama-3.1-8B-Instruct |
|
- **Quantization**: 4-bit AWQ |
|
- **Engine**: LMDeploy TurboMindEngine |
|
|
|
|
|
```bash |
|
lmdeploy lite auto_awq \ |
|
$HF_MODEL \ |
|
--calib-dataset 'ptb' \ |
|
--calib-samples 128 \ |
|
--calib-seqlen 2048 \ |
|
--w-bits 4 \ |
|
--w-group-size 128 \ |
|
--batch-size 10 \ |
|
--search-scale True \ |
|
--work-dir $WORK_DIR |
|
``` |
