metadata

base_model: upstage/SOLAR-10.7B-Instruct-v1.0
inference: false
language:
  - en
license: apache-2.0
model-index:
  - name: SOLAR-10.7B-Instruct-v1.0
    results: []
model_creator: Upstage
model_name: SOLAR-10.7B-Instruct-v1.0
model_type: solar
prompt_template: |
  <|im_start|>system
  {system_message}<|im_end|>
  <|im_start|>user
  {prompt}<|im_end|>
  <|im_start|>assistant
quantized_by: Inferless
tags:
  - SOLAR
  - instruct
  - finetune
  - vllm
  - GPTQ

Serverless GPUs to scale your machine learning inference without any hassle of managing servers, deploy complicated and custom models with ease.

Join Private Beta

SOLAR-10.7B-Instruct-v1.0 - GPTQ

Model creator: Upstage
Original model: SOLAR-10.7B-Instruct-v1.0

Description

This repo contains GPTQ model files for Upstage's SOLAR-10.7B-Instruct-v1.0.

About GPTQ

GPTQ is a method that compresses the model size and accelerates inference by quantizing weights based on a calibration dataset, aiming to minimize mean squared error in a single post-quantization step. GPTQ achieves both memory efficiency and faster inference.

It is supported by:

Text Generation Webui - using Loader: AutoAWQ
vLLM - version 0.2.2 or later for support for all model types.
Hugging Face Text Generation Inference (TGI)
Transformers version 4.35.0 and later, from any code or client that supports Transformers
AutoAWQ - for use from Python code

Provided files, and AWQ parameters

I currently release 128g GEMM models only. The addition of group_size 32 models, and GEMV kernel models, is being actively considered.

Models are released as sharded safetensors files.

Branch	Bits	GS	AWQ Dataset	Seq Len	Size
main	4	128	VMware Open Instruct	4096	5.96 GB