SGEcon/EconFinKoSOLAR-10.7B_SFT

Model Details

Model Developers: Sogang University SGEconFinlab(<https://sc.sogang.ac.kr/aifinlab/)

Model Description

This model is a language model specialized in economics and finance. This was learned with various economic/finance-related data. The data sources are listed below, and we are not releasing the data that we trained on because it was used for research/policy purposes. If you wish to use the original data, please contact the original author directly for permission to use it.

• Developed by: Sogang University SGEconFinlab(https://sc.sogang.ac.kr/aifinlab/)
• License: cc-by-nc-4.0
• Base Model: yanolja/KoSOLAR-10.7B-v0.2(https://huggingface.co/yanolja/KoSOLAR-10.7B-v0.2)

Loading the Model

peft_model_id = "SGEcon/EconFinKoSOLAR-10.7B_SFT"
config = PeftConfig.from_pretrained(peft_model_id)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, quantization_config=bnb_config, device_map={"":0})
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
model.eval()

Conducting Conversation

import re

def gen(x):
    inputs = tokenizer(f"### 질문: {x}\n\n### 답변:", return_tensors='pt', return_token_type_ids=False)

    # Move data to GPU (if available)
    inputs = {k: v.to(device="cuda" if torch.cuda.is_available() else "cpu") for k, v in inputs.items()}

    gened = model.generate(
        **inputs,
        max_new_tokens=256,  # Maximum number of new tokens to create
        early_stopping=True,
        num_return_sequences=1,  # Generate only one answer
        do_sample=True,  # Enable sampling to generate a variety of answers
        eos_token_id=tokenizer.eos_token_id,  # Using EOS Token IDs 
        temperature=0.9,  # This option is adjustable.
        top_p=0.8,  # This option is adjustable.
        top_k=100  # This option is adjustable.
    )

    # Decode the generated sequence and convert it to output text 
    decoded = tokenizer.decode(gened[0], skip_special_tokens=True).strip()

    # Extract only text after a string "### 답변:" 
    answer_start_idx = decoded.find("### 답변:") + len("### 답변:")
    complete_answer = decoded[answer_start_idx:].strip()

    # Find the first punctuation mark (. ? !) and extract only up to it
    match = re.search(r"[\.\?\!][^\.\?\!]*$", complete_answer)
    if match:
        complete_answer = complete_answer[:match.end()].strip()

    return complete_answer

Training Details

• We train our model with PEFT. PEFT is a technique that does not tune all parameters of a model during fine-tuning, but only a small subset of parameters. By tuning only a few parameters while leaving others fixed, the model is less likely to suffer from catastrophic forgetting, where the model forgets previously learned tasks when it learns new ones. This significantly reduces computation and storage costs.

• We use QLora to train the base model. Quantized Low Rank Adapters (QLoRA) is an efficient technique that uses 4-bit quantized pre-trained language models to fine-tune 65 billion parameter models on a 48 GB GPU while significantly reducing memory usage. The method uses NormalFloat 4-bit (NF4), a new data type that is theoretically optimal for normally distributed weights; Double Quantization, which further quantizes quantization constants to reduce average memory usage; and Paged Optimizers, which manage memory spikes during mini-batch processing, to increase memory efficiency without sacrificing performance.

• Also, we performed instruction tuning using the data that we collected and the kyujinpy/KOR-OpenOrca-Platypus-v3 dataset on the hugging face. Instruction tuning is learning in a supervised learning format that uses instructions and input data together as input and output data as a pair. In other words, instruction tuning involves fine-tuning a pre-trained model for a specific task or set of tasks, where the model is taught to follow specific instructions or guidelines. Instruction tuning is a type of Supervised Fine-tuning (SFT) that aims to improve the generality and adaptability of a model by introducing an additional dimension that enables the model to understand and follow specific instructions.

Training Data

1. 한국은행: 경제금융용어 700선(https://www.bok.or.kr/portal/bbs/B0000249/view.do?nttId=235017&menuNo=200765)
2. 금융감독원: 금융소비자 정보 포털 파인 금융용어사전(https://fine.fss.or.kr/fine/fnctip/fncDicary/list.do?menuNo=900021)
3. KDI 경제정보센터: 시사 용어사전(https://eiec.kdi.re.kr/material/wordDic.do)
4. 한국경제신문/한경닷컴: 한경경제용어사전(https://terms.naver.com/list.naver?cid=42107&categoryId=42107), 오늘의 TESAT(https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=1), 오늘의 주니어 TESAT(https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=5), 생글생글한경(https://sgsg.hankyung.com/tesat/study)
5. 중소벤처기업부/대한민국정부: 중소벤처기업부 전문용어(https://terms.naver.com/list.naver?cid=42103&categoryId=42103)
6. 고성삼/법문출판사: 회계·세무 용어사전(https://terms.naver.com/list.naver?cid=51737&categoryId=51737)
7. 맨큐의 경제학 8판 Word Index
8. kyujinpy/KOR-OpenOrca-Platypus-v3(https://huggingface.co/datasets/kyujinpy/KOR-OpenOrca-Platypus-v3)

At the request of the original author, it is not to be used for commercial purposes. Therefore, it is licensed under the license CC-BY-NC-4.0. The copyright of the data used belongs to the original author, so please contact the original author when using it.

Training Hyperparameters

Hyperparameter	SGEcon/KoSOLAR-10.7B-v0.2_fin_v4
Lora Method	Lora
load in 4 bit	True
learning rate	1e-5
lr scheduler	linear
lora alpa	16
lora rank	16
lora dropout	0.05
optim	paged_adamw_32bit
target_modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, lm_head

License

The language identification model is distributed under the Creative Commons Attribution-NonCommercial 4.0 International Public License.

Example

We only removed duplicate sentences.

중앙은행의 역할에 대해서 설명해줄래?

중앙은행은 국가가 발행하는 화폐의 유통과 경제의 안정적 운영을 위해 설립된 국가기관입니다. 중앙은행은 ① 경제의 안정적 운영을 위해 금융기관을 감독하고, 화폐의 발행과 통화정책을 실시하는 기능을 합니다. ② 통화정책의 실행에는 중앙은행이 유동성(Liquidity)을 공급하고, 이자율을 조정하여 수요와 공급을 조절하는 일, 예금준비금을 유지하는 일, 환율조절 등이 포함됩니다. ③ 중앙은행은 금융기관의 대출한도 초과대출, 외환거래, 지폐의 발행과 교환을 관리하는 업무도 담당합니다. ④ 통화정책의 실시는 화폐와 신용의 발행 및 공급을 통제를 통해 물가안정과 고용을 증대하고 경제의 균형성장을 도모하는 것을 목표로 하고 있습니다.