ISTA-DASLab

/

Meta-Llama-3.1-70B-Instruct-AQLM-PV-2Bit-1x16

like 43

Follow

IST Austria Distributed Algorithms and Systems Lab 46

Using the latest version of vllm 0.6.1.post2 throws Unsupported base layer: QKVParallelLinear(in_features=8192, output_features=10240, bias=False, tp_size=1, gather_output=False) during init