|
--- |
|
language: |
|
- ru |
|
tags: |
|
- PyTorch |
|
- Transformers |
|
thumbnail: "https://github.com/sberbank-ai/ru-gpts" |
|
--- |
|
|
|
# rugpt3xl |
|
Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048. |
|
*Note! Model has sparse attention blocks.* |
|
|
|
Total training time was around 10 days on 256 GPUs. |
|
Final perplexity on test set is `12.05`. |
|
Model parameters: 1.3B. |
|
from transformers import GPT2LMHeadModel, GPT2Tokenizer |
|
model_name_or_path = "sberbank-ai/rugpt3large_based_on_gpt2" (можно использовать sberbank-ai/rugpt3xl) |
|
tokenizer = GPT2Tokenizer.from_pretrained(model_name_or_path) |
|
model = GPT2LMHeadModel.from_pretrained(model_name_or_path).cpu() |
|
text = "Иисус Христос родился в " |
|
input_ids = tokenizer.encode(text, return_tensors="pt").cpu() |
|
out = model.generate(input_ids.cpu()) |
|
print(generated_text) |
|
generated_text = list(map(tokenizer.decode, out))[0] |
|
print(generated_text) |
|
|
|
|
