For this quantization, we used 1 codebook of 16 bits.

Results:

Model	Quantization	MMLU (5-shot)	ArcC	ArcE	Hellaswag	PiQA	Winogrande	Model size, Gb
meta-llama/Meta-Llama-3-8B	None	0.6525	0.5043	0.8009	0.6019	0.7971	0.7261	16.1
	1x16	0.5647	0.4130	0.7428	0.5545	0.7780	0.7198	4.1

UPD 02.05.2024

The version of model with improved fine-tuning procedure.

Safetensors

Model size

2.04B params

Tensor type

FP16

I16

Inference Examples

Inference API (serverless) is not available, repository is disabled.

Collection including ISTA-DASLab/Meta-Llama-3-8B-AQLM-2Bit-1x16