File size: 3,051 Bytes
1bf3599 1aff62d 1bf3599 1aff62d 1bf3599 d7a97db 1bf3599 1aff62d 1bf3599 1aff62d 1bf3599 1aff62d 1bf3599 7367820 1aff62d 1bf3599 1aff62d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
library_name: transformers
tags:
- SkillTree
- llama2
license: llama2
---
# SkillTree Model Collection
Applying a skill to your model with SkillTree is akin to unlocking a new ability in a video game's skill tree. Just as you would enhance your character's capabilities by selecting and activating specific skills, you can augment your model's abilities by integrating specialized skills. Follow these steps to imbue your model with new prowess, enhancing its performance and versatility in a straightforward and intuitive manner.
**Please note that SkillTree abilities may not function in all cases. To determine whether a specific skill is operational, refer to the Functionality Status.**
## What is SkillTree?
SkillTree represents a set of model weights derived from further pre-training or fine-tuning Large Language Models (LLMs) to extract specific capabilities, such as code generation or chatting abilities. These extracted "skills" can be combined with a specific LLM base model to enhance its capabilities. The concept is inspired by [ChatVector](https://arxiv.org/abs/2310.04799), aiming to modularize and transfer distinct skills across models.
## SkillTree Details
- **Functionality Status:** Functional / **Non-Functional** / Not Verified
- **Base Model:** [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
- **Skill Model:** [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)
- **Enhanced Model(optional):** [HachiML/Swallow-7b-hf-CodeSkill](https://huggingface.co/HachiML/Swallow-7b-hf-CodeSkill)
- **Skill type:** Coding
## Uses
### Limitation
- **Model Architecture:** Llama2
- **Model Size:** 6.74B
- **Compatible Models[optional]:**
### How to Apply Skill (Example)
```python
# Import Library
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# Load the target model to be applied skill
tokenizer = AutoTokenizer.from_pretrained(
"tokyotech-llm/Swallow-7b-hf"
)
model = AutoModelForCausalLM.from_pretrained(
"tokyotech-llm/Swallow-7b-hf",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Load SkillTree
skill_tree = AutoModelForCausalLM.from_pretrained(
"HachiML/SkillTree-llama2-7b-hf-Code",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Apply the skill to the target model
def apply_skill(model, skill_tree):
# excluded object
skip_layers = ["model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"]
# apply skill
for k, v in model.state_dict().items():
# layernorm is also excluded
if (k in skip_layers) or ("layernorm" in k):
continue
vector = skill_tree.state_dict()[k]
new_v = v + vector.to(v.device)
v.copy_(new_v)
return model
model = apply_skill(model, skill_tree)
# Push to hub
model_name = "HachiML/Swallow-7b-hf-CodeSkill"
tokenizer.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
model.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
``` |