Francesco-A commited on
Commit
669ef59
1 Parent(s): f7848c8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -9,9 +9,9 @@ The Code Search Net Tokenizer is a custom tokenizer specifically trained for tok
9
 
10
  **Model Details:**
11
 
12
- Name: Code Search Net Tokenizer
13
- Model Type: Custom Tokenizer
14
- Language: Python
15
 
16
  **Training Data:**
17
 
@@ -19,9 +19,9 @@ The tokenizer was trained on a corpus of Python code snippets from the CodeSearc
19
 
20
  **Tokenizer Features:**
21
 
22
- *The Code Search Net Tokenizer offers the following features:
23
 
24
- *Tokenization of Python code: The tokenizer can effectively split Python code snippets into individual tokens, making it suitable for downstream tasks that involve code processing and understanding.
25
 
26
  **Usage:**
27
 
 
9
 
10
  **Model Details:**
11
 
12
+ - Name: Code Search Net Tokenizer
13
+ - Model Type: Custom Tokenizer
14
+ - Language: Python
15
 
16
  **Training Data:**
17
 
 
19
 
20
  **Tokenizer Features:**
21
 
22
+ - The Code Search Net Tokenizer offers the following features:
23
 
24
+ - Tokenization of Python code: The tokenizer can effectively split Python code snippets into individual tokens, making it suitable for downstream tasks that involve code processing and understanding.
25
 
26
  **Usage:**
27