| Model Info (Interal): | |
| - Size: 7B | |
| - Dataset: The Pile v2 | |
| - `contaminated(P3) + lower_code(5%) + wiki(fixed) + books3(fixed & broken)` | |
| - Batch size (in tokens): 8M | |
| - Checkpoint Step: 69,000 (552B tokens) | |
| - Checkpoint path (AWS East): `/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf` | |
| Notes: | |
| - Trained for 36k steps with incorrectly tokenized Books3 dataset (GPT-2 tokenizer instead of NeoX tokenizer) | |
| - tp=2 (not 4) | |
| W&B Report: https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-7B-alpha---Vmlldzo2MjA | |
| Usage: | |
| ```python | |
| import transformers | |
| model = transformers.AutoModelForCausalLM.from_pretrained("CarperAI/7b-alpha") | |
| tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b") | |
| tokenizer.pad_token = tokenizer.eos_token | |
| tokenizer.paddding_side = "left" | |
| prompts = [ | |
| "User1: The dog sat on a man's lap and barked 3 times.\nUser2: How many times did the dog bark?" | |
| "Curious Person Question: A group of genetically identical individuals is called what?\nSmart Person Answer: a clone\n\nCurious Person Question: Who proposed the theory of evolution by natural selection?\nSmart Person Answer:" | |
| ] | |
| batch_encoding = tokenizer(prompts, return_tensors="pt", padding=True) | |
| print(f"Generating {len(prompts)} prompts...") | |
| samples = model.generate( | |
| **batch_encoding, | |
| max_new_tokens=64, | |
| temperature=0.0, | |
| do_sample=False, | |
| ) | |
| samples = tokenizer.batch_decode(samples, skip_special_tokens=True) | |
| for prompt, sample in zip(prompts, samples): | |
| print(f"Prompt: {prompt}\nSample: {sample}\n") | |
| ``` |