metadata

libray_name: transformers
pipeline_tag: text-generation
license: other
license_name: llama3
license_link: LICENSE
language:
  - ko
  - en
tags:
  - meta
  - llama
  - llama-3
  - akallama
library_name: transformers

AKALLAMA

AkaLlama is a series of Korean language models designed for practical usability across a wide range of tasks. The initial model, AkaLlama-v0.1, is a fine-tuned version of Meta-Llama-3-70b-Instruct. It has been trained on a custom mix of publicly available datasets curated by the MIR Lab. Our goal is to explore cost-effective ways to adapt high-performing LLMs for specific use cases, such as different languages (e.g., Korean) or domains (e.g., organization-specific chatbots).

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub.

Developed by: Yonsei MIRLab
Language(s) (NLP): Korean, English
License: llama3
Finetuned from model: meta-llama/Meta-Llama-3-70B-Instruct

How to use

This repo provides full model weight files for AkaLlama-70B-v0.1.

Use with transformers

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "mirlab/AkaLlama-llama3-70b-v0.1"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="auto",
)

system_prompt = """
"""

messages = [
    {"role": "system", "content": "system_prompt"},
    {"role": "user", "content": "네 이름은 뭐야?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])

Training Details

Training Procedure

We trained AkaLlama using a preference learning alignment algorithm called Odds Ratio Preference Optimization (ORPO). Our training pipeline is almost identical to that of HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1, aside from minor hyperparameter changes. Please check out Huggingface's alignment handbook for further details, including the chat template.

Training Data

Detailed descriptions regarding training data will be announced later.

Examples

Math Solving[CLICK TO EXPAND]

Writting[CLICK TO EXPAND]

logical Reasoning[CLICK TO EXPAND]

Coding [CLICK TO EXPAND]

You can find more examples at our project page

Special Thanks

Data Center of the Department of Artificial Intelligence at Yonsei University for the computation resources