base_model: []
library_name: transformers
tags:
- mergekit
- merge
Midnight-Miqu-70B-v1.5 - EXL2 2.75bpw
This is a 2.75bpw EXL2 quant of sophosympatheia/Midnight-Miqu-70B-v1.5
Details about the model and the merge info can be found at the above mode page.
I have not extensively tested this quant/model other than ensuring I could load it and chat with it.
Tavern Card
Included is a Tavern format character card created by Midnight Miqu v1.5 for chat. The card was created using a character creator helper bot using a single prompt for the base card, another prompt asking for specific conversation examples and then asking it to provide a text to image portrait prompt. Being able to faithfully follow the character creator bot to create this card demonstrates a pretty high level of intelligence.
An ageless, charismatic archmage named Elrondel stands before you, captured mid-laugh in a dimly lit study. His tall, slender frame is adorned with midnight blue robes that billow around him, intricate silver runes glinting in the candlelight. His stark white hair flows down to his shoulders, framing a sharp face with piercing sapphire eyes that seem to look right through you. A knowing smile plays upon his lips, hinting at countless secrets hidden beneath his pointed goatee.
In one hand, he holds a smoldering pipe filled with exotic herbs, casting a mysterious haze around his form. The other hand rests upon Stormbreaker, a colossal staff that towers above most men, crackling with latent lightning energy. Various rings adorn his long, dexterous fingers, each holding its own arcane power. Around his neck hangs a dragon-eye amulet, the eye itself appearing to follow your every move with fierce scrutiny.
The room is cluttered with tomes and scrolls, floating in midair as if held by invisible hands, creating a maelstrom of knowledge. Behind him, a crystal ball reflects swirling images of distant lands and times, while a cauldron bubbles with unknown concoctions on the hearth of an ancient fireplace. The scene exudes an air of enigma and might, leaving you both awestruck and slightly intimidated in the presence of this legendary figure from Thaylonia.
Quant Details
This is the script used for quantization.
#!/bin/bash
# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2
# Define variables
MODEL_DIR="models/Midnight-Miqu-70B-v1.5"
OUTPUT_DIR="exl2_midnightv15-70b"
MEASUREMENT_FILE="measurements/midnight70b-v15.json"
TEMPLATE_FILE="models/Midnight-Miqu-70B-v1.5/README-TEMPLATE.md"
BIT_PRECISIONS=(6.0 5.0 4.5 4.0 3.5 3.0 2.75 2.5 2.25)
for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
CONVERTED_FOLDER="models/Midnight-Miqu-70B-v1.5_exl2_${BIT_PRECISION}bpw"
UPLOAD_FOLDER="Dracones/Midnight-Miqu-70B-v1.5_exl2_${BIT_PRECISION}bpw"
if [ -d "$CONVERTED_FOLDER" ]; then
echo "Skipping $BIT_PRECISION as $CONVERTED_FOLDER already exists."
continue
fi
rm -r "$OUTPUT_DIR"
mkdir "$OUTPUT_DIR"
mkdir "$CONVERTED_FOLDER"
python convert.py -i "$MODEL_DIR" -o "$OUTPUT_DIR" -nr -m "$MEASUREMENT_FILE" -b "$BIT_PRECISION" -cf "$CONVERTED_FOLDER"
cp "$TEMPLATE_FILE" "$CONVERTED_FOLDER/README.md"
sed -i "s/X.XX/$BIT_PRECISION/g" "$CONVERTED_FOLDER/README.md"
done