Hugging Face's logo

speakleash
/
Bielik-11B-v2.3-Instruct

Text Generation
Transformers
Safetensors
Polish
mistral
Merge
mergekit
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
Edit model card

Bielik-11B-v2.3-Instruct

Bielik-11B-v2.3-Instruct is a generative text model featuring 11 billion parameters. It is a linear merge of the Bielik-11B-v2.0-Instruct, Bielik-11B-v2.1-Instruct, and Bielik-11B-v2.2-Instruct models, which are instruct fine-tuned versions of the Bielik-11B-v2. Forementioned model stands as a testament to the unique collaboration between the open-science/open-souce project SpeakLeash and the High Performance Computing (HPC) center: ACK Cyfronet AGH. Developed and trained on Polish text corpora, which has been cherry-picked and processed by the SpeakLeash team, this endeavor leverages Polish large-scale computing infrastructure, specifically within the PLGrid environment, and more precisely, the HPC centers: ACK Cyfronet AGH. The creation and training of the Bielik-11B-v2.3-Instruct was propelled by the support of computational grant number PLG/2024/016951, conducted on the Athena and Helios supercomputer, enabling the use of cutting-edge technology and computational resources essential for large-scale machine learning processes. As a result, the model exhibits an exceptional ability to understand and process the Polish language, providing accurate responses and performing a variety of linguistic tasks with high precision.

🗣️ Chat Arena*: https://arena.speakleash.org.pl/

*Chat Arena is a platform for testing and comparing different AI language models, allowing users to evaluate their performance and quality.

Model

The SpeakLeash team is working on their own set of instructions in Polish, which is continuously being expanded and refined by annotators. A portion of these instructions, which had been manually verified and corrected, has been utilized for training purposes. Moreover, due to the limited availability of high-quality instructions in Polish, synthetic instructions were generated with Mixtral 8x22B and used in training. The dataset used for training comprised over 20 million instructions, consisting of more than 10 billion tokens. The instructions varied in quality, leading to a deterioration in the model’s performance. To counteract this while still allowing ourselves to utilize the aforementioned datasets, several improvements were introduced:

To align the model with user preferences we tested many different techniques: DPO, PPO, KTO, SiMPO. Finally the DPO-Positive method was employed, utilizing both generated and manually corrected examples, which were scored by a metamodel. A dataset comprising over 66,000 examples of varying lengths to address different aspects of response style. It was filtered and evaluated by the reward model to select instructions with the right level of difference between chosen and rejected. The novelty introduced in DPO-P was multi-turn conversations introduction.

Bielik instruct models have been trained with the use of an original open source framework called ALLaMo implemented by Krzysztof Ociepa. This framework allows users to train language models with architecture similar to LLaMA and Mistral in fast and efficient way.

Bielik-11B-v2.3-Instruct is a merge of the Bielik-11B-v2.0-Instruct, Bielik-11B-v2.1-Instruct, and Bielik-11B-v2.2-Instruct models. The merge was performed in float16 precision by Remigiusz Kinas using mergekit.

Model description:

Quantized models:

We know that some people want to explore smaller models or don't have the resources to run a full model. Therefore, we have prepared quantized versions of the Bielik-11B-v2.3-Instruct model in separate repositories:

Please note that quantized models may offer lower quality of generated answers compared to full sized variatns.

Chat template

Bielik-11B-v2.3-Instruct uses ChatML as the prompt format.

E.g.

prompt = "<s><|im_start|> user\nJakie mamy pory roku?<|im_end|> \n<|im_start|> assistant\n"
completion = "W Polsce mamy 4 pory roku: wiosna, lato, jesień i zima.<|im_end|> \n"

This format is available as a chat template via the apply_chat_template() method:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model_name = "speakleash/Bielik-11B-v2.3-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

messages = [
    {"role": "system", "content": "Odpowiadaj krótko, precyzyjnie i wyłącznie w języku polskim."},
    {"role": "user", "content": "Jakie mamy pory roku w Polsce?"},
    {"role": "assistant", "content": "W Polsce mamy 4 pory roku: wiosna, lato, jesień i zima."},
    {"role": "user", "content": "Która jest najcieplejsza?"}
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = input_ids.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

Fully formated input conversation by apply_chat_template from previous example:

<s><|im_start|> system
Odpowiadaj krótko, precyzyjnie i wyłącznie w języku polskim.<|im_end|> 
<|im_start|> user
Jakie mamy pory roku w Polsce?<|im_end|> 
<|im_start|> assistant
W Polsce mamy 4 pory roku: wiosna, lato, jesień i zima.<|im_end|> 
<|im_start|> user
Która jest najcieplejsza?<|im_end|>

Evaluation

Bielik-11B-v2.3-Instruct has been evaluated on several benchmarks to assess its performance across various tasks and languages. These benchmarks include:

  1. Open PL LLM Leaderboard
  2. Open LLM Leaderboard
  3. Polish MT-Bench
  4. Polish EQ-Bench (Emotional Intelligence Benchmark)
  5. MixEval

The following sections provide detailed results for each of these benchmarks, demonstrating the model's capabilities in both Polish and English language tasks.

Open PL LLM Leaderboard

Models have been evaluated on Open PL LLM Leaderboard 5-shot. The benchmark evaluates models in NLP tasks like sentiment analysis, categorization, text classification but does not test chatting skills. Average column is an average score among all tasks normalized by baseline scores.

Model Parameters (B) Average
Meta-Llama-3.1-405B-Instruct-FP8,API 405 69.44
Mistral-Large-Instruct-2407 123 69.11
Qwen2-72B-Instruct 72 65.87
Bielik-11B-v2.3-Instruct 11 65.71
Bielik-11B-v2.2-Instruct 11 65.57
Meta-Llama-3.1-70B-Instruct 70 65.49
Bielik-11B-v2.1-Instruct 11 65.45
Mixtral-8x22B-Instruct-v0.1 141 65.23
Bielik-11B-v2.0-Instruct 11 64.98
Meta-Llama-3-70B-Instruct 70 64.45
Athene-70B 70 63.65
WizardLM-2-8x22B 141 62.35
Qwen1.5-72B-Chat 72 58.67
Qwen2-57B-A14B-Instruct 57 56.89
glm-4-9b-chat 9 56.61
aya-23-35B 35 56.37
Phi-3.5-MoE-instruct 41.9 56.34
openchat-3.5-0106-gemma 7 55.69
Mistral-Nemo-Instruct-2407 12 55.27
SOLAR-10.7B-Instruct-v1.0 10.7 55.24
Mixtral-8x7B-Instruct-v0.1 46.7 55.07
Bielik-7B-Instruct-v0.1 7 44.70
trurl-2-13b-academic 13 36.28
trurl-2-7b 7 26.93

The results from the Open PL LLM Leaderboard demonstrate the exceptional performance of Bielik-11B-v2.3-Instruct:

  1. Superior performance in its class: Bielik-11B-v2.3-Instruct outperforms all other models with less than 70B parameters. This is a significant achievement, showcasing its efficiency and effectiveness despite having fewer parameters than many competitors.

  2. Competitive with larger models: with a score of 65.71, Bielik-11B-v2.3-Instruct performs on par with models in the 70B parameter range. This indicates that it achieves comparable results to much larger models, demonstrating its advanced architecture and training methodology.

  3. Substantial improvement over previous version: the model shows a marked improvement over its predecessor, Bielik-7B-Instruct-v0.1, which scored 43.64. This leap in performance highlights the successful enhancements and optimizations implemented in this newer version.

  4. Leading position for Polish language models: in the context of Polish language models, Bielik-11B-v2.3-Instruct stands out as a leader. There are no other competitive models specifically tailored for the Polish language that match its performance, making it a crucial resource for Polish NLP tasks.

These results underscore Bielik-11B-v2.3-Instruct's position as a state-of-the-art model for Polish language processing, offering high performance with relatively modest computational requirements.

Open PL LLM Leaderboard - Generative Tasks Performance

This section presents a focused comparison of generative Polish language task performance between Bielik models and GPT-3.5. The evaluation is limited to generative tasks due to the constraints of assessing OpenAI models. The comprehensive nature and associated costs of the benchmark explain the limited number of models evaluated.

Model Parameters (B) Average g
Bielik-11B-v2.3-Instruct 11 67.47
Bielik-11B-v2.1-Instruct 11 66.58
Bielik-11B-v2.2-Instruct 11 66.11
Bielik-11B-v2.0-Instruct 11 65.58
gpt-3.5-turbo-instruct Unknown 55.65

The performance variation among Bielik versions is minimal, indicating consistent quality across iterations. Bielik-11B-v2.3-Instruct demonstrates an impressive 21.2% performance advantage over GPT-3.5.

Open LLM Leaderboard

The Open LLM Leaderboard evaluates models on various English language tasks, providing insights into the model's performance across different linguistic challenges.

Model AVG arc_challenge hellaswag truthfulqa_mc2 mmlu winogrande gsm8k
Bielik-11B-v2.2-Instruct 69.86 59.90 80.16 58.34 64.34 75.30 81.12
Bielik-11B-v2.3-Instruct 69.82 59.30 80.11 57.42 64.57 76.24 81.27
Bielik-11B-v2.1-Instruct 69.82 59.56 80.20 59.35 64.18 75.06 80.59
Bielik-11B-v2.0-Instruct 68.04 58.62 78.65 54.65 63.71 76.32 76.27
Bielik-11B-v2 65.87 60.58 79.84 46.13 63.06 77.82 67.78
Mistral-7B-Instruct-v0.2 65.71 63.14 84.88 68.26 60.78 77.19 40.03
Bielik-7B-Instruct-v0.1 51.26 47.53 68.91 49.47 46.18 65.51 29.95

Bielik-11B-v2.3-Instruct shows impressive performance on English language tasks:

  1. Significant improvement over its base model (4-point increase).
  2. Substantial 18-point improvement over Bielik-7B-Instruct-v0.1.

These results demonstrate Bielik-11B-v2.3-Instruct's versatility in both Polish and English, highlighting the effectiveness of its instruction tuning process.

Polish MT-Bench

The Bielik-11B-v2.3-Instruct (16 bit) model was also evaluated using the MT-Bench benchmark. The quality of the model was evaluated using the English version (original version without modifications) and the Polish version created by Speakleash (tasks and evaluation in Polish, the content of the tasks was also changed to take into account the context of the Polish language).

MT-Bench English

Model Score
Bielik-11B-v2.1 8.537500
Bielik-11B-v2.3 8.531250
Bielik-11B-v2.2 8.390625
Bielik-11B-v2.0 8.159375

MT-Bench Polish

Model Parameters (B) Score
Qwen2-72B-Instruct 72 8.775000
Mistral-Large-Instruct-2407 (123B) 123 8.662500
gemma-2-27b-it 27 8.618750
Bielik-11B-v2.3-Instruct 11 8.556250
Mixtral-8x22b 141 8.231250
Meta-Llama-3.1-405B-Instruct 405 8.168750
Meta-Llama-3.1-70B-Instruct 70 8.150000
Bielik-11B-v2.2-Instruct 11 8.115625
Bielik-11B-v2.1-Instruct 11 7.996875
gpt-3.5-turbo Unknown 7.868750
Mixtral-8x7b 46.7 7.637500
Bielik-11B-v2.0-Instruct 11 7.562500
Mistral-Nemo-Instruct-2407 12 7.368750
openchat-3.5-0106-gemma 7 6.812500
Mistral-7B-Instruct-v0.2 7 6.556250
Meta-Llama-3.1-8B-Instruct 8 6.556250
Bielik-7B-Instruct-v0.1 7 6.081250
Mistral-7B-Instruct-v0.3 7 5.818750
Polka-Mistral-7B-SFT 7 4.518750
trurl-2-7b 7 2.762500

Key observations on Bielik-11B-v2.3 performance:

  1. Strong performance among mid-sized models: Bielik-11B-v2.3-Instruct scored 8.556250, placing it ahead of several well-known models like GPT-3.5-turbo (7.868750) and Mixtral-8x7b (7.637500). This indicates that Bielik-11B-v2.3-Instruct is competitive among mid-sized models, particularly those in the 11B-70B parameter range.

  2. Competitive against larger models: Bielik-11B-v2.3-Instruct performs close to Meta-Llama-3.1-70B-Instruct (8.150000), Meta-Llama-3.1-405B-Instruct (8.168750) and even Mixtral-8x22b (8.231250), which have significantly more parameters. This efficiency in performance relative to size could make it an attractive option for tasks where resource constraints are a consideration. Bielik 100% generated answers in Polish, while other models (not typically trained for Polish) can answer Polish questions in English.

  3. Significant improvement over previous versions: compared to its predecessor, Bielik-7B-Instruct-v0.1, which scored 6.081250, the Bielik-11B-v2.3-Instruct shows a significant improvement. The score increased by almost 2.5 points, highlighting substantial advancements in model quality, optimization and training methodology.

For more information - answers to test tasks and values in each category, visit the MT-Bench PL website.

Polish EQ-Bench

Polish Emotional Intelligence Benchmark for LLMs

Model Parameters (B) Score
Mistral-Large-Instruct-2407 123 78.07
Meta-Llama-3.1-405B-Instruct-FP8 405 77.23
gpt-4o-2024-08-06 ? 75.15
gpt-4-turbo-2024-04-09 ? 74.59
Meta-Llama-3.1-70B-Instruct 70 72.53
Qwen2-72B-Instruct 72 71.23
Meta-Llama-3-70B-Instruct 70 71.21
gpt-4o-mini-2024-07-18 ? 71.15
Bielik-11B-v2.3-Instruct 11 70.86
WizardLM-2-8x22B 141 69.56
Bielik-11B-v2.2-Instruct 11 69.05
Bielik-11B-v2.0-Instruct 11 68.24
Qwen1.5-72B-Chat 72 68.03
Mixtral-8x22B-Instruct-v0.1 141 67.63
Bielik-11B-v2.1-Instruct 11 60.07
Qwen1.5-32B-Chat 32 59.63
openchat-3.5-0106-gemma 7 59.58
aya-23-35B 35 58.41
gpt-3.5-turbo ? 57.7
Qwen2-57B-A14B-Instruct 57 57.64
Mixtral-8x7B-Instruct-v0.1 47 57.61
SOLAR-10.7B-Instruct-v1.0 10.7 55.21
Mistral-7B-Instruct-v0.2 7 47.02

MixEval

MixEval is a ground-truth-based English benchmark designed to evaluate Large Language Models (LLMs) efficiently and effectively. Key features of MixEval include:

  1. Derived from off-the-shelf benchmark mixtures
  2. Highly capable model ranking with a 0.96 correlation to Chatbot Arena
  3. Local and quick execution, requiring only 6% of the time and cost compared to running MMLU

This benchmark provides a robust and time-efficient method for assessing LLM performance, making it a valuable tool for ongoing model evaluation and comparison.

Model MixEval MixEval-Hard
Bielik-11B-v2.1-Instruct 74.55 45.00
Bielik-11B-v2.3-Instruct 72.95 43.20
Bielik-11B-v2.2-Instruct 72.35 39.65
Bielik-11B-v2.0-Instruct 72.10 40.20
Mistral-7B-Instruct-v0.2 70.00 36.20

The results show that Bielik-11B-v2.3-Instruct performs well on the MixEval benchmark, achieving a score of 72.95 on the standard MixEval and 43.20 on MixEval-Hard. Notably, Bielik-11B-v2.3-Instruct significantly outperforms Mistral-7B-Instruct-v0.2 on both metrics, demonstrating its improved capabilities despite being based on a similar architecture.

Limitations and Biases

Bielik-11B-v2.3-Instruct is a quick demonstration that the base model can be easily fine-tuned to achieve compelling and promising performance. It does not have any moderation mechanisms. We're looking forward to engaging with the community in ways to make the model respect guardrails, allowing for deployment in environments requiring moderated outputs.

Bielik-11B-v2.3-Instruct can produce factually incorrect output, and should not be relied on to produce factually accurate data. Bielik-11B-v2.3-Instruct was trained on various public datasets. While great efforts have been taken to clear the training data, it is possible that this model can generate lewd, false, biased or otherwise offensive outputs.

Citation

Please cite this model using the following format:

@misc{Bielik11Bv21i,
    title     = {Bielik-11B-v2.3-Instruct model card},
    author    = {Ociepa, Krzysztof and Flis, Łukasz and Kinas, Remigiusz and Gwoździej, Adrian and Wróbel, Krzysztof and {SpeakLeash Team} and {Cyfronet Team}},
    year      = {2024},
    url       = {https://huggingface.co/speakleash/Bielik-11B-v2.3-Instruct},
    note      = {Accessed: 2024-09-16}, % change this date
    urldate   = {2024-09-16} % change this date
}
@unpublished{Bielik11Bv23a,
  author = {Ociepa, Krzysztof and Flis, Łukasz and Kinas, Remigiusz and Gwoździej, Adrian and Wróbel, Krzysztof},
  title  = {Bielik: A Family of Large Language Models for the Polish Language - Development, Insights, and Evaluation},
  year   = {2024},
}
@misc{ociepa2024bielik7bv01polish,
      title={Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation}, 
      author={Krzysztof Ociepa and Łukasz Flis and Krzysztof Wróbel and Adrian Gwoździej and Remigiusz Kinas},
      year={2024},
      eprint={2410.18565},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.18565}, 
}

Responsible for training the model

  • Krzysztof OciepaSpeakLeash - team leadership, conceptualizing, data preparation, process optimization and oversight of training
  • Łukasz FlisCyfronet AGH - coordinating and supervising the training
  • Remigiusz KinasSpeakLeash - conceptualizing and coordinating DPO training, data preparation
  • Adrian GwoździejSpeakLeash - data preparation and ensuring data quality
  • Krzysztof WróbelSpeakLeash - benchmarks

The model could not have been created without the commitment and work of the entire SpeakLeash team, whose contribution is invaluable. Thanks to the hard work of many individuals, it was possible to gather a large amount of content in Polish and establish collaboration between the open-science SpeakLeash project and the HPC center: ACK Cyfronet AGH. Individuals who contributed to the creation of the model: Sebastian Kondracki, Igor Ciuciura, Paweł Kiszczak, Szymon Baczyński, Jacek Chwiła, Maria Filipkowska, Jan Maria Kowalski, Karol Jezierski, Kacper Milan, Jan Sowa, Len Krawczyk, Marta Seidler, Agnieszka Ratajska, Krzysztof Koziarek, Szymon Pepliński, Zuzanna Dabić, Filip Bogacz, Agnieszka Kosiak, Izabela Babis, Nina Babis.

Members of the ACK Cyfronet AGH team providing valuable support and expertise: Szymon Mazurek, Marek Magryś, Mieszko Cholewa .

Contact Us

If you have any questions or suggestions, please use the discussion tab. If you want to contact us directly, join our Discord SpeakLeash.

Downloads last month
9,211
Safetensors
Model size
11.2B params
Tensor type
FP16
·
Inference Examples
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for speakleash/Bielik-11B-v2.3-Instruct

speakleash/Bielik-11B-v2
speakleash/Bielik-11B-v2.0-Instruct
speakleash/Bielik-11B-v2.1-Instruct
speakleash/Bielik-11B-v2.2-Instruct
Merge model
this model
Finetunes
3 models
Quantizations
8 models

Space using speakleash/Bielik-11B-v2.3-Instruct 1

Collection including speakleash/Bielik-11B-v2.3-Instruct