kushal-tri commited on
Commit
aa945dc
1 Parent(s): 3cb87dd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +146 -3
README.md CHANGED
@@ -1,3 +1,146 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+
5
+
6
+
7
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/63118add64939fabc0108b28/BB42g4V8HTxb5dR4tcy8A.png" alt="DCLM Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
8
+
9
+
10
+ # Model Card for DCLM-1B-IT
11
+
12
+ DCLM-IT-1B is a 1.4B billion parameter language model trained on the DCLM-Baseline dataset and then further finetuned on our DCLM-IT finetuning mixture. This model is designed to showcase the effectiveness of systematic data curation techniques for improving language model performance.
13
+
14
+ ## Model Details
15
+
16
+ | Size | Training Tokens | Layers | Hidden Size | Attention Heads | Context Length |
17
+ |:------:|:-----------------:|:--------:|:-------------:|:-----------------:|:----------------:|
18
+ | 1.4B | 2.608T | 24 | 2048 | 16 | 2048 |
19
+
20
+
21
+ ### Model Description
22
+
23
+ - **Developed by:** DataComp for Language Models (DCLM) Team
24
+ - **Model type:** Decoder-only Transformer language model
25
+ - **Language(s):** English (primarily)
26
+ - **License:** Apache 2.0
27
+ - **Contact:** [email protected]
28
+ - **Date:** July 2024
29
+
30
+ ### Model Sources
31
+
32
+ - **Repository:** https://github.com/mlfoundations/dclm
33
+ - **Dataset:** https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0
34
+ - **Paper:** [DataComp-LM: In search of the next generation of training sets for language models](https://arxiv.org/abs/2406.11794)
35
+
36
+
37
+
38
+ ### Instruction Tuning Details
39
+
40
+ The model was trained using the following setup:
41
+
42
+ - **Architecture:** Decoder-only Transformer
43
+ - **Framework:** PyTorch with OpenLM
44
+ - **Optimizer:** AdamW
45
+ - **Learning Rate:** 2e-5 (peak)
46
+ - **Weight Decay:** 0.1
47
+ - **Batch Size:** 2048 sequences
48
+ - **Sequence Length:** 2048 tokens
49
+ - **Total Training Tokens:** 8.4B
50
+ - **Number of Epochs**: 10
51
+ - **Hardware:** Trained on H100 GPUs
52
+
53
+ For more detailed training information, please refer to Section 3.4 and Appendix F of the DCLM paper.
54
+
55
+ ## Evaluation
56
+ Here are the evaluation results for DCLM-1B models on various tasks (using [llm-foundry](https://github.com/mosaicml/llm-foundry) eval suite)
57
+
58
+ | Task | Core | Extended | MMLU 5-shot |
59
+ |:---------:|:------:|:----------:|:-------------:|
60
+ | DCLM-1B | 42.3 | 25.1 | 41.9 |
61
+ | DCLM-1B-v2| 45.2 | 28.1 | 47.5 |
62
+ | DCLM-1B-IT| 47.1 | 33.6 | 51.4 |
63
+
64
+ Note: All scores are presented as decimal values between 0 and 1, representing the proportion of correct answers or the model's performance on each task.
65
+
66
+ Moreover, we present our evaluation results on Length-Controlled Alpaca-Eval 2.0 to measure our instruction-following capabilities.
67
+
68
+ | Model | AlpacaEval2.0 LC Win-rate (%) |
69
+ |------------------------------------|------------------------------:|
70
+ | **Our runs** | |
71
+ | DCLM-IT-1B | 8.6 |
72
+ | DCLM-IT-7B | 16.6 |
73
+ | Mistral-7B w/ OpenHermes 2.5 | 15.4 |
74
+ | DCLM-Baseline-7B w/ OpenHermes 2.5 | 13.8 |
75
+ | **Reported from the leaderboard** | |
76
+ | LLaMA-3-Instruct-8B | **22.9** |
77
+ | Mistral-v0.2-7B | 17.1 |
78
+ | Mistral-7B w/ OpenHermes 2.5 | 16.2 |
79
+ | Zephyr-Beta-7B | 13.2 |
80
+ | Vicuna-v1.3-13B | 10.8 |
81
+ | Gemma-Instruct-7B | 10.4 |
82
+ | Nous-Hermes-13B | 9.7 |
83
+ | DaVinci001 | 9.0 |
84
+ | LLaMA-2-Chat-13B | 8.4 |
85
+ | Alpaca-7B | 5.9 |
86
+
87
+ ## Example Code
88
+
89
+ This is example code on how to run the chat model.
90
+ ```
91
+ from transformers import AutoTokenizer
92
+ from open_lm.utils.transformers.hf_config import OpenLMConfig
93
+ import torch
94
+ from open_lm.utils.transformers.hf_model import OpenLMConfig, OpenLMforCausalLM
95
+
96
+ # Load the model and tokenizer
97
+ model_name = "TRI-ML/DCLM-1B-IT"
98
+ # Load the configuration, tokenizer, and model separately
99
+ config = OpenLMConfig.from_pretrained(model_name)
100
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
101
+ model = OpenLMforCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="cuda", config=config)
102
+
103
+ # Define the prompt format
104
+ def create_prompt(instruction):
105
+ PROMPT = '''Below is an instruction that describes a task.\n\nWrite a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:'''
106
+ return PROMPT.format(instruction=instruction)
107
+ # Example instruction
108
+ instruction = "Give me a poem about Sachin Tendulkar."
109
+ # Create the prompt
110
+ prompt = create_prompt(instruction)
111
+ # Tokenize the input
112
+ input_ids = tokenizer.encode(prompt, return_tensors="pt").to(torch.device('cuda'))
113
+ # Generate the response
114
+ output = model.generate(input_ids, max_length=500, top_p=.95, do_sample=True, temperature=0.3)
115
+ # Decode the response
116
+ response = tokenizer.decode(output[0][len(input_ids[0]):])
117
+ response = response.split("<|endoftext|>")[0]
118
+ # Print the response
119
+ print(response)
120
+ ```
121
+
122
+ ## Limitations and Biases
123
+
124
+ While DCLM-1B demonstrates strong performance across a range of tasks, it's important to note:
125
+
126
+ 1. The model may exhibit biases present in its training data, which is derived from web crawl data.
127
+ 2. It has not undergone specific alignment or safety fine-tuning, so outputs should be used with caution.
128
+ 3. Performance on tasks not included in the evaluation suite may vary.
129
+ 4. The model's knowledge is limited to its training data cutoff date.
130
+
131
+ ## Ethical Considerations
132
+
133
+ Users should be aware that this model, like all large language models, can potentially generate harmful or biased content. It should not be used for making decisions about individuals or in sensitive applications without appropriate safeguards and human oversight.
134
+
135
+ ## Citation
136
+
137
+ If you use this model in your research, please cite:
138
+
139
+ ```
140
+ @article{Li2024DataCompLM,
141
+ title={DataComp-LM: In search of the next generation of training sets for language models},
142
+ author={Jeffrey Li and Alex Fang and Georgios Smyrnis and Maor Ivgi and Matt Jordan and Samir Gadre and Hritik Bansal and Etash Guha and Sedrick Keh and Kushal Arora and [... full author list]},
143
+ journal={arXiv preprint arXiv:2406.11794},
144
+ year={2024}
145
+ }
146
+ ```