jordimas commited on
Commit
8cdb89c
1 Parent(s): fda88c7

Update documentation

Browse files
Files changed (3) hide show
  1. README.md +55 -3
  2. config.json +2 -0
  3. model_description.txt +9 -0
README.md CHANGED
@@ -1,3 +1,55 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ca
4
+ - fr
5
+
6
+ tags:
7
+ - translation
8
+
9
+ library_name: opennmt
10
+ license: mit
11
+ metrics:
12
+ - bleu
13
+
14
+ inference: false
15
+ ---
16
+
17
+ ### Introduction
18
+
19
+ French - Catalan translation model based on OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
20
+
21
+
22
+ ### Usage
23
+
24
+
25
+ ```bash
26
+ pip3 install ctranslate2 pyonmttok
27
+ ```
28
+
29
+ Simple translation using Python:
30
+
31
+
32
+
33
+ ```python
34
+ import ctranslate2
35
+ import pyonmttok
36
+ from huggingface_hub import snapshot_download
37
+ model_dir = snapshot_download(repo_id="softcatala/translate-fra-cat", revision="main")
38
+
39
+ tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
40
+ tokenized=tokenizer.tokenize("Salut monde")
41
+
42
+ import ctranslate2
43
+ translator = ctranslate2.Translator(model_dir)
44
+ translated = translator.translate_batch([tokenized[0]])
45
+ print(tokenizer.detokenize(translated[0][0]['tokens']))
46
+ Hola món!
47
+ ```
48
+
49
+ ## Benchmarks
50
+
51
+ | testset | BLEU |
52
+ |---------------------------------------|-------|
53
+ | test dataset (from train/dev/test) | 41.3 |
54
+ | Flores200 dataset | 31.6 |
55
+
config.json ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ {
2
+ }
model_description.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ Model description: fra-cat
2
+ Date: 2022-11-09
3
+ TF version 2.10.0, OpenNMT version 2.29.1, CTranslate2 version 2.24.0
4
+ Test data set
5
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 41.3 68.6/47.8/36.0/27.6 (BP = 0.971 ratio = 0.972 hyp_len = 152857 ref_len = 157316)
6
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 64.1
7
+ Flores data set
8
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 31.6 61.6/37.8/25.1/17.2 (BP = 1.000 ratio = 1.006 hyp_len = 27470 ref_len = 27304)
9
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 58.7