Update README.md
Browse files
README.md
CHANGED
|
@@ -3,6 +3,7 @@ Model Info (Interal):
|
|
| 3 |
- Dataset: The Pile v2
|
| 4 |
- `contaminated(P3) + lower_code(5%) + wiki(fixed) + books3(fixed & broken)`
|
| 5 |
- Batch size (in tokens): 8M
|
|
|
|
| 6 |
- Checkpoint path (AWS East): `/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf`
|
| 7 |
|
| 8 |
Notes:
|
|
|
|
| 3 |
- Dataset: The Pile v2
|
| 4 |
- `contaminated(P3) + lower_code(5%) + wiki(fixed) + books3(fixed & broken)`
|
| 5 |
- Batch size (in tokens): 8M
|
| 6 |
+
- Checkpoint Step: 69,000 (552B tokens)
|
| 7 |
- Checkpoint path (AWS East): `/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf`
|
| 8 |
|
| 9 |
Notes:
|