![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Igc5itf-xMGj"
},
"source": [
"# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "x4fXCKCf36IK"
},
"source": [
"## Note before beginning:\n",
"### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
"\n",
"### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
"\n",
"### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
"\n",
"### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
"\n",
"### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
"\n",
"### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "l929HimrxS0a"
},
"source": [
"## Retrieve your data & make a parallel corpus\n",
"\n",
"If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
"\n",
"Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "oGRmDELn7Az0",
"outputId": "56d0cd12-61f6-4a4b-d2b8-74abfff2f815",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 127
}
},
"source": [
"from google.colab import drive\n",
"drive.mount('/content/drive')"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
"\n",
"Enter your authorization code:\n",
"··········\n",
"Mounted at /content/drive\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "Cn3tgQLzUxwn",
"colab": {}
},
"source": [
"# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
"# These will also become the suffix's of all vocab and corpus files used throughout\n",
"import os\n",
"source_language = \"yo\"\n",
"target_language = \"en\"\n",
"lc = False # If True, lowercase the data.\n",
"seed = 42 # Random seed for shuffling.\n",
"tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
"\n",
"os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
"os.environ[\"tgt\"] = target_language\n",
"os.environ[\"tag\"] = tag\n",
"\n",
"# This will save it to a folder in our gdrive instead! \n",
"!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
"g_drive_path = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)\n",
"os.environ[\"gdrive_path\"] = g_drive_path\n",
"models_path = '%s/models/%s%s_transformer'% (g_drive_path, source_language, target_language)\n",
"# model temporary directory for training\n",
"model_temp_dir = \"/content/drive/My Drive/masakhane/model-temp\"\n",
"# model permanent storage on the drive\n",
"!mkdir -p \"$gdrive_path/models/${src}${tgt}_transformer/\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "kBSgJHEw7Nvx",
"outputId": "6e14d6cf-9290-4fc3-cb32-3deb7224255a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"!echo $gdrive_path"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"/content/drive/My Drive/masakhane/yo-en-baseline\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "gA75Fs9ys8Y9",
"outputId": "83fb02a5-bd6b-4b83-b155-078ae07ded95",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 102
}
},
"source": [
"#TODO: Skip for retrain\n",
"# Install opus-tools\n",
"! pip install opustools-pkg "
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting opustools-pkg\n",
"\u001b[?25l Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
"\u001b[K |████████████████████████████████| 81kB 9.5MB/s \n",
"\u001b[?25hInstalling collected packages: opustools-pkg\n",
"Successfully installed opustools-pkg-0.0.52\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "xq-tDZVks7ZD",
"outputId": "12ccab59-4d5d-4e28-8163-4a39fe17f233",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 221
}
},
"source": [
"#TODO: Skip for retrain\n",
"# Downloading our corpus\n",
"! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
"\n",
"# extract the corpus file\n",
"! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
"Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-yo.xml.gz not found. The following files are available for downloading:\n",
"\n",
" 4 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-yo.xml.gz\n",
" 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
" 58 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/yo.zip\n",
"\n",
" 325 MB Total size\n",
"./JW300_latest_xml_en-yo.xml.gz ... 100% of 4 MB\n",
"./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
"./JW300_latest_xml_yo.zip ... 100% of 58 MB\n",
"gzip: JW300_latest_xml_yo-en.xml.gz: No such file or directory\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "doraUei7W31h",
"colab_type": "code",
"colab": {}
},
"source": [
"# extract the corpus file\n",
"! gunzip JW300_latest_xml_$tgt-$src.xml.gz"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "n48GDRnP8y2G",
"colab_type": "code",
"outputId": "4f30b67f-6d04-4dd9-fa2a-859eb5a076b2",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 578
}
},
"source": [
"#TODO: Skip for retrain\n",
"# Download the global test set.\n",
"! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
" \n",
"# And the specific test set for this language pair.\n",
"os.environ[\"trg\"] = target_language \n",
"os.environ[\"src\"] = source_language \n",
"\n",
"! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$src.en \n",
"! mv test.en-$src.en test.en\n",
"! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$src.$src \n",
"! mv test.en-$src.$src test.$src"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"--2020-04-07 20:43:18-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 277791 (271K) [text/plain]\n",
"Saving to: ‘test.en-any.en’\n",
"\n",
"\rtest.en-any.en 0%[ ] 0 --.-KB/s \rtest.en-any.en 100%[===================>] 271.28K --.-KB/s in 0.008s \n",
"\n",
"2020-04-07 20:43:19 (32.5 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
"\n",
"--2020-04-07 20:43:22-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-yo.en\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 201994 (197K) [text/plain]\n",
"Saving to: ‘test.en-yo.en’\n",
"\n",
"test.en-yo.en 100%[===================>] 197.26K --.-KB/s in 0.005s \n",
"\n",
"2020-04-07 20:43:22 (36.5 MB/s) - ‘test.en-yo.en’ saved [201994/201994]\n",
"\n",
"--2020-04-07 20:43:27-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-yo.yo\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 280073 (274K) [text/plain]\n",
"Saving to: ‘test.en-yo.yo’\n",
"\n",
"test.en-yo.yo 100%[===================>] 273.51K --.-KB/s in 0.007s \n",
"\n",
"2020-04-07 20:43:29 (36.3 MB/s) - ‘test.en-yo.yo’ saved [280073/280073]\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "NqDG-CI28y2L",
"colab_type": "code",
"outputId": "6a52eb39-aff5-41b8-d427-aa128cf73e76",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"#TODO: Skip for retrain\n",
"# Read the test data to filter from train and dev splits.\n",
"# Store english portion in set for quick filtering checks.\n",
"en_test_sents = set()\n",
"filter_test_sents = \"test.en-any.en\"\n",
"j = 0\n",
"with open(filter_test_sents) as f:\n",
" for line in f:\n",
" en_test_sents.add(line.strip())\n",
" j += 1\n",
"print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Loaded 3571 global test sentences to filter from the training/dev data.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "3CNdwLBCfSIl",
"outputId": "c7e16fbb-ba42-41b2-b73b-49df100de222",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 159
}
},
"source": [
"#TODO: Skip for retrain\n",
"import pandas as pd\n",
"\n",
"# TMX file to dataframe\n",
"source_file = 'jw300.' + source_language\n",
"target_file = 'jw300.' + target_language\n",
"\n",
"source = []\n",
"target = []\n",
"skip_lines = [] # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
"with open(source_file) as f:\n",
" for i, line in enumerate(f):\n",
" # Skip sentences that are contained in the test set.\n",
" if line.strip() not in en_test_sents:\n",
" source.append(line.strip())\n",
" else:\n",
" skip_lines.append(i) \n",
"with open(target_file) as f:\n",
" for j, line in enumerate(f):\n",
" # Only add to corpus if corresponding source was not skipped.\n",
" if j not in skip_lines:\n",
" target.append(line.strip())\n",
" \n",
"print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
" \n",
"df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
"# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
"#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
"df.head(3)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Loaded data and skipped 1025/474986 lines since contained in test set.\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/html": [
"| \n", " | source_sentence | \n", "target_sentence | \n", "
|---|---|---|
| 0 | \n", "Lílo Àkàbà — Ǹjẹ́ O Máa Ń Ṣe Àyẹ̀wò Wọ̀nyí Tó... | \n", "Using Ladders — Do You Make These Safety Checks ? | \n", "
| 1 | \n", "Látọwọ́ akọ̀ròyìn Jí ! | \n", "By Awake ! | \n", "
| 2 | \n", "ní Ireland | \n", "correspondent in Ireland | \n", "