|
--- |
|
language: |
|
- en |
|
license: apache-2.0 |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- unsloth |
|
- llama |
|
- trl |
|
- sft |
|
base_model: meta-llama/Meta-Llama-3-8B |
|
datasets: |
|
- teknium/OpenHermes-2.5 |
|
- grimulkan/theory-of-mind |
|
- grimulkan/physical-reasoning |
|
--- |
|
|
|
# Llama3 8B Wordcel |
|
|
|
Wordcel is a Llama3 fine-tune intended to be used as a mid-training checkpoint for more specific RP/storywriting/creative applications. |
|
|
|
It has been trained from Llama3 8B Base on a composite dataset of ~100M tokens that highlights reasoning, (uncensored) stories, classic literature, and assorted interpersonal intelligence tasks. |
|
|
|
Components of the composite dataset include [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5), and [Grimulkan](https://huggingface.co/grimulkan)'s [Theory of Mind](https://huggingface.co/datasets/grimulkan/theory-of-mind) and [Physical Reasoning](https://huggingface.co/datasets/grimulkan/physical-reasoning) datasets. |
|
|
|
It is trained at a context length of 32k tokens, using linear RoPE scaling with a factor of 4.0. Derivative models should be capable of generalizing to 32k tokens as a result. |
|
|
|
If you train a model using this checkpoint, please give clear attribution! The Llama 3 base license likely applies. |
