Francesco-A
commited on
Commit
•
669ef59
1
Parent(s):
f7848c8
Update README.md
Browse files
README.md
CHANGED
@@ -9,9 +9,9 @@ The Code Search Net Tokenizer is a custom tokenizer specifically trained for tok
|
|
9 |
|
10 |
**Model Details:**
|
11 |
|
12 |
-
Name: Code Search Net Tokenizer
|
13 |
-
Model Type: Custom Tokenizer
|
14 |
-
Language: Python
|
15 |
|
16 |
**Training Data:**
|
17 |
|
@@ -19,9 +19,9 @@ The tokenizer was trained on a corpus of Python code snippets from the CodeSearc
|
|
19 |
|
20 |
**Tokenizer Features:**
|
21 |
|
22 |
-
|
23 |
|
24 |
-
|
25 |
|
26 |
**Usage:**
|
27 |
|
|
|
9 |
|
10 |
**Model Details:**
|
11 |
|
12 |
+
- Name: Code Search Net Tokenizer
|
13 |
+
- Model Type: Custom Tokenizer
|
14 |
+
- Language: Python
|
15 |
|
16 |
**Training Data:**
|
17 |
|
|
|
19 |
|
20 |
**Tokenizer Features:**
|
21 |
|
22 |
+
- The Code Search Net Tokenizer offers the following features:
|
23 |
|
24 |
+
- Tokenization of Python code: The tokenizer can effectively split Python code snippets into individual tokens, making it suitable for downstream tasks that involve code processing and understanding.
|
25 |
|
26 |
**Usage:**
|
27 |
|