Hugging Face's logo

mhhmm
/
typescript-instruct-20k-v2

Text Generation
PEFT
PyTorch
TensorBoard
English
llama
typescript
instruction-tuning
code-generation
lora
Model card Files Metrics Community
mhhmm commited on
Commit
4bfa8b8
•
1 Parent(s): fb332d6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -44
README.md CHANGED
@@ -2,36 +2,33 @@
2
  license: llama2
3
  library_name: peft
4
  tags:
5
- - generated_from_trainer
 
 
 
 
6
  base_model: codellama/CodeLlama-13b-hf
7
  model-index:
8
  - name: lora-out
9
  results: []
 
 
 
 
 
 
 
10
  ---
11
 
12
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
- should probably proofread and complete it, then remove this comment. -->
14
 
15
- [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
16
- # lora-out
17
 
18
- This model is a fine-tuned version of [codellama/CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf) on the None dataset.
19
- It achieves the following results on the evaluation set:
20
- - Loss: 0.4263
21
-
22
- ## Model description
23
-
24
- More information needed
25
-
26
- ## Intended uses & limitations
27
-
28
- More information needed
29
-
30
- ## Training and evaluation data
31
-
32
- More information needed
33
 
34
- ## Training procedure
 
 
35
 
36
  ### Training hyperparameters
37
 
@@ -53,26 +50,27 @@ The following hyperparameters were used during training:
53
 
54
  | Training Loss | Epoch | Step | Validation Loss |
55
  |:-------------:|:-----:|:----:|:---------------:|
56
- | 0.7628 | 0.01 | 1 | 0.7296 |
57
- | 0.7101 | 0.05 | 7 | 0.6906 |
58
- | 0.5395 | 0.1 | 14 | 0.5214 |
59
- | 0.5303 | 0.15 | 21 | 0.4871 |
60
- | 0.4821 | 0.2 | 28 | 0.4676 |
61
- | 0.5643 | 0.25 | 35 | 0.4563 |
62
- | 0.5307 | 0.3 | 42 | 0.4484 |
63
- | 0.5103 | 0.35 | 49 | 0.4445 |
64
- | 0.5515 | 0.4 | 56 | 0.4415 |
65
- | 0.4983 | 0.45 | 63 | 0.4386 |
66
- | 0.4919 | 0.5 | 70 | 0.4351 |
67
- | 0.4674 | 0.55 | 77 | 0.4316 |
68
- | 0.5193 | 0.6 | 84 | 0.4295 |
69
- | 0.4461 | 0.65 | 91 | 0.4295 |
70
- | 0.4541 | 0.71 | 98 | 0.4280 |
71
- | 0.486 | 0.76 | 105 | 0.4280 |
72
- | 0.4875 | 0.81 | 112 | 0.4269 |
73
- | 0.5553 | 0.86 | 119 | 0.4266 |
74
- | 0.4605 | 0.91 | 126 | 0.4260 |
75
- | 0.4767 | 0.96 | 133 | 0.4263 |
 
76
 
77
 
78
  ### Framework versions
@@ -81,10 +79,27 @@ The following hyperparameters were used during training:
81
  - Pytorch 2.0.1+cu118
82
  - Datasets 2.15.0
83
  - Tokenizers 0.15.0
84
- ## Training procedure
 
 
85
 
 
86
 
87
- ### Framework versions
88
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
 
90
- - PEFT 0.6.0
 
2
  license: llama2
3
  library_name: peft
4
  tags:
5
+ - typescript
6
+ - instruction-tuning
7
+ - code-generation
8
+ - lora
9
+ - peft
10
  base_model: codellama/CodeLlama-13b-hf
11
  model-index:
12
  - name: lora-out
13
  results: []
14
+ datasets:
15
+ - mhhmm/typescript-instruct-20k
16
+ language:
17
+ - en
18
+ metrics:
19
+ - code_eval
20
+ pipeline_tag: text-generation
21
  ---
22
 
23
+ ## Architecture
 
24
 
25
+ ![The Architecture](https://github.com/LeVuMinhHuy/brocode/blob/master/.pics/about-the-model.png?raw=true)
 
26
 
27
+ ## About
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
 
29
+ This model is a fine-tuned version of [codellama/CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf).
30
+ It achieves the following results on the evaluation set:
31
+ - Loss: 0.4268
32
 
33
  ### Training hyperparameters
34
 
 
50
 
51
  | Training Loss | Epoch | Step | Validation Loss |
52
  |:-------------:|:-----:|:----:|:---------------:|
53
+ | 0.7555 | 0.01 | 1 | 0.7062 |
54
+ | 0.7036 | 0.05 | 7 | 0.6673 |
55
+ | 0.5422 | 0.1 | 14 | 0.5152 |
56
+ | 0.5351 | 0.15 | 21 | 0.4866 |
57
+ | 0.495 | 0.2 | 28 | 0.4688 |
58
+ | 0.5651 | 0.25 | 35 | 0.4587 |
59
+ | 0.5146 | 0.3 | 42 | 0.4486 |
60
+ | 0.4955 | 0.35 | 49 | 0.4469 |
61
+ | 0.5117 | 0.4 | 56 | 0.4432 |
62
+ | 0.5245 | 0.45 | 63 | 0.4410 |
63
+ | 0.5003 | 0.5 | 70 | 0.4371 |
64
+ | 0.4502 | 0.55 | 77 | 0.4340 |
65
+ | 0.527 | 0.6 | 84 | 0.4315 |
66
+ | 0.48 | 0.65 | 91 | 0.4305 |
67
+ | 0.448 | 0.7 | 98 | 0.4289 |
68
+ | 0.5427 | 0.75 | 105 | 0.4289 |
69
+ | 0.4715 | 0.8 | 112 | 0.4279 |
70
+ | 0.5584 | 0.85 | 119 | 0.4276 |
71
+ | 0.4936 | 0.9 | 126 | 0.4267 |
72
+ | 0.4788 | 0.95 | 133 | 0.4268 |
73
+ | 0.476 | 1.0 | 140 | 0.4268 |
74
 
75
 
76
  ### Framework versions
 
79
  - Pytorch 2.0.1+cu118
80
  - Datasets 2.15.0
81
  - Tokenizers 0.15.0
82
+ - PEFT 0.6.0
83
+
84
+ ### Evaluation
85
 
86
+ I'm using MultiPL-E benchmark, the same as Code Llmama using in their paper
87
 
88
+ How to reproduce my evaluation? Just run like the offical document of MultiPL-E: https://nuprl.github.io/MultiPL-E/tutorial.html, change the modal name by my model here: `mhhmm/typescript-instruct-20k`
89
 
90
+ This is the code that I ran with Google Colab (using A100 40GB, yes, it requires that much GPU RAM)
91
+
92
+ If you even have a stronger GPU, increase the --batch-size, or --completion-limit
93
+
94
+ ```
95
+ !pip install --upgrade pip
96
+ !pip install aiohttp numpy tqdm pytest datasets torch transformers sentencepiece
97
+ !git clone https://github.com/nuprl/MultiPL-E
98
+ %cd MultiPL-E
99
+ !mkdir typescript
100
+ !python3 automodel.py --name mhhmm/typescript-instruct-20k --root-dataset humaneval --lang ts --temperature 0.2 --batch-size 10 --completion-limit 20 --output-dir-prefix typescript
101
+ %cd evaluation/src
102
+ !python3 main.py --dir ../../typescript --output-dir ../../typescript --recursive
103
+ !python3 pass_k.py ./typescript/*
104
+ ```
105