en-st/jw300-baseline/readme.md

English to Sesotho

Author: Jason Webster

A en_st_process.ipynb and en_st_train_from_drive.ipynb notebook are present in order to be able to run the processing and training steps at two different times. The fuzzywuzzy preprocessing step took long enough to consume a full Google Colab instance's time allocation, so a seperate notebook was needed in order to train the model from the preprocessed data. The en_st_train_from_drive.ipynb notebook also contains code for testing on the Autshumato dataset, though these results are reported in a seperate folder.

Training Data

- The JW300 dataset.

Preprocessing

- Removed duplicate sentences
- Removed similar sentences with fuzzywuzzy and a similarity score above 0.95
- 40 000 BPE codes used

Model

- Default Masakhane Transformer translation model (see `en_st_train_from_drive.ipynb` for detailed config)
- Link to google drive folder with model (https://drive.google.com/drive/folders/1--C5IwyI_P0B4_-nAsBDaBgrGqm5ABvY?usp=sharing)
- Trained for approx 9h30min

Analysis

 - TODO

Results

- JW300 `test.en` and `test.st`
- BLEU dev: 46.15
- BLEU test: 41.23

- Note: It is probably best to train this model for longer, as it was close to timing out on Google Colab (was manually stopped)
- Note: Will likely benefit from optimising the number of BPE codes