julien-c HF staff commited on
Commit
a46768e
1 Parent(s): 9b9b744

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/huggingface/CodeBERTa-small-v1/README.md

Files changed (1) hide show
  1. README.md +130 -0
README.md ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: code
3
+ thumbnail: https://cdn-media.huggingface.co/CodeBERTa/CodeBERTa.png
4
+ datasets:
5
+ - code_search_net
6
+ ---
7
+
8
+ # CodeBERTa
9
+
10
+ CodeBERTa is a RoBERTa-like model trained on the [CodeSearchNet](https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/) dataset from GitHub.
11
+
12
+ Supported languages:
13
+
14
+ ```shell
15
+ "go"
16
+ "java"
17
+ "javascript"
18
+ "php"
19
+ "python"
20
+ "ruby"
21
+ ```
22
+
23
+ The **tokenizer** is a Byte-level BPE tokenizer trained on the corpus using Hugging Face `tokenizers`.
24
+
25
+ Because it is trained on a corpus of code (vs. natural language), it encodes the corpus efficiently (the sequences are between 33% to 50% shorter, compared to the same corpus tokenized by gpt2/roberta).
26
+
27
+ The (small) **model** is a 6-layer, 84M parameters, RoBERTa-like Transformer model – that’s the same number of layers & heads as DistilBERT – initialized from the default initialization settings and trained from scratch on the full corpus (~2M functions) for 5 epochs.
28
+
29
+ ### Tensorboard for this training ⤵️
30
+
31
+ [![tb](https://cdn-media.huggingface.co/CodeBERTa/tensorboard.png)](https://tensorboard.dev/experiment/irRI7jXGQlqmlxXS0I07ew/#scalars)
32
+
33
+ ## Quick start: masked language modeling prediction
34
+
35
+ ```python
36
+ PHP_CODE = """
37
+ public static <mask> set(string $key, $value) {
38
+ if (!in_array($key, self::$allowedKeys)) {
39
+ throw new \InvalidArgumentException('Invalid key given');
40
+ }
41
+ self::$storedValues[$key] = $value;
42
+ }
43
+ """.lstrip()
44
+ ```
45
+
46
+ ### Does the model know how to complete simple PHP code?
47
+
48
+ ```python
49
+ from transformers import pipeline
50
+
51
+ fill_mask = pipeline(
52
+ "fill-mask",
53
+ model="huggingface/CodeBERTa-small-v1",
54
+ tokenizer="huggingface/CodeBERTa-small-v1"
55
+ )
56
+
57
+ fill_mask(PHP_CODE)
58
+
59
+ ## Top 5 predictions:
60
+ #
61
+ ' function' # prob 0.9999827146530151
62
+ 'function' #
63
+ ' void' #
64
+ ' def' #
65
+ ' final' #
66
+ ```
67
+
68
+ ### Yes! That was easy 🎉 What about some Python (warning: this is going to be meta)
69
+
70
+ ```python
71
+ PYTHON_CODE = """
72
+ def pipeline(
73
+ task: str,
74
+ model: Optional = None,
75
+ framework: Optional[<mask>] = None,
76
+ **kwargs
77
+ ) -> Pipeline:
78
+ pass
79
+ """.lstrip()
80
+ ```
81
+
82
+ Results:
83
+ ```python
84
+ 'framework', 'Framework', ' framework', 'None', 'str'
85
+ ```
86
+
87
+ > This program can auto-complete itself! 😱
88
+
89
+ ### Just for fun, let's try to mask natural language (not code):
90
+
91
+ ```python
92
+ fill_mask("My name is <mask>.")
93
+
94
+ # {'sequence': '<s> My name is undefined.</s>', 'score': 0.2548016905784607, 'token': 3353}
95
+ # {'sequence': '<s> My name is required.</s>', 'score': 0.07290805131196976, 'token': 2371}
96
+ # {'sequence': '<s> My name is null.</s>', 'score': 0.06323737651109695, 'token': 469}
97
+ # {'sequence': '<s> My name is name.</s>', 'score': 0.021919190883636475, 'token': 652}
98
+ # {'sequence': '<s> My name is disabled.</s>', 'score': 0.019681859761476517, 'token': 7434}
99
+ ```
100
+
101
+ This (kind of) works because code contains comments (which contain natural language).
102
+
103
+ Of course, the most frequent name for a Computer scientist must be undefined 🤓.
104
+
105
+
106
+ ## Downstream task: [programming language identification](https://huggingface.co/huggingface/CodeBERTa-language-id)
107
+
108
+ See the model card for **[`huggingface/CodeBERTa-language-id`](https://huggingface.co/huggingface/CodeBERTa-language-id)** 🤯.
109
+
110
+ <br>
111
+
112
+ ## CodeSearchNet citation
113
+
114
+ <details>
115
+
116
+ ```bibtex
117
+ @article{husain_codesearchnet_2019,
118
+ title = {{CodeSearchNet} {Challenge}: {Evaluating} the {State} of {Semantic} {Code} {Search}},
119
+ shorttitle = {{CodeSearchNet} {Challenge}},
120
+ url = {http://arxiv.org/abs/1909.09436},
121
+ urldate = {2020-03-12},
122
+ journal = {arXiv:1909.09436 [cs, stat]},
123
+ author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
124
+ month = sep,
125
+ year = {2019},
126
+ note = {arXiv: 1909.09436},
127
+ }
128
+ ```
129
+
130
+ </details>