{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "accelerator": "GPU", "colab": { "name": "starter_notebook.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.6" } }, "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Igc5itf-xMGj" }, "source": [ "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "x4fXCKCf36IK" }, "source": [ "## Note before beginning:\n", "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n", "\n", "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n", "\n", "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n", "\n", "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n", "\n", "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n", "\n", "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "l929HimrxS0a" }, "source": [ "## Retrieve your data & make a parallel corpus\n", "\n", "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n", "\n", "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. " ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "oGRmDELn7Az0", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "b8270ce4-6e57-4ce8-e7d9-46d20290dd69" }, "source": [ "from google.colab import drive\n", "drive.mount('/content/drive')" ], "execution_count": 26, "outputs": [ { "output_type": "stream", "text": [ "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "Cn3tgQLzUxwn", "colab": {} }, "source": [ "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n", "# These will also become the suffix's of all vocab and corpus files used throughout\n", "import os\n", "source_language = \"en\"\n", "target_language = \"urh\" \n", "lc = False # If True, lowercase the data.\n", "seed = 42 # Random seed for shuffling.\n", "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n", "\n", "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n", "os.environ[\"tgt\"] = target_language\n", "os.environ[\"tag\"] = tag\n", "\n", "# This will save it to a folder in our gdrive instead!\n", "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n", "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "kBSgJHEw7Nvx", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "d786cba5-3631-4959-8875-c15ef993c21e" }, "source": [ "!echo $gdrive_path" ], "execution_count": 28, "outputs": [ { "output_type": "stream", "text": [ "/content/drive/My Drive/masakhane/en-urh-baseline\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "gA75Fs9ys8Y9", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "04370262-f35c-4d29-b251-41d182537b7f" }, "source": [ "# Install opus-tools\n", "! pip install opustools-pkg" ], "execution_count": 29, "outputs": [ { "output_type": "stream", "text": [ "Requirement already satisfied: opustools-pkg in /usr/local/lib/python3.6/dist-packages (0.0.52)\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "xq-tDZVks7ZD", "colab": { "base_uri": "https://localhost:8080/", "height": 187 }, "outputId": "b62cc8ce-a08b-494f-ea6d-d262f7d2c6f8" }, "source": [ "# Downloading our corpus\n", "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n", "\n", "# extract the corpus file\n", "! gunzip JW300_latest_xml_$src-$tgt.xml.gz" ], "execution_count": 30, "outputs": [ { "output_type": "stream", "text": [ "\n", "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-urh.xml.gz not found. The following files are available for downloading:\n", "\n", " ./JW300_latest_xml_en.zip already exists\n", " ./JW300_latest_xml_urh.zip already exists\n", " 304 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-urh.xml.gz\n", "\n", " 304 KB Total size\n", "./JW300_latest_xml_en-urh.xml.gz ... 100% of 304 KB\n", "gzip: JW300_latest_xml_en-urh.xml already exists; do you wish to overwrite (y or n)? y\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "n48GDRnP8y2G", "colab_type": "code", "colab": { "base_uri": "https://localhost:8080/", "height": 578 }, "outputId": "b50339aa-aae7-4027-a985-f303c50b290b" }, "source": [ "# Download the global test set.\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n", " \n", "# And the specific test set for this language pair.\n", "os.environ[\"trg\"] = target_language \n", "os.environ[\"src\"] = source_language \n", "\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n", "! mv test.en-$trg.en test.en\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n", "! mv test.en-$trg.$trg test.$trg" ], "execution_count": 31, "outputs": [ { "output_type": "stream", "text": [ "--2019-12-31 06:41:57-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 277791 (271K) [text/plain]\n", "Saving to: ‘test.en-any.en.1’\n", "\n", "\rtest.en-any.en.1 0%[ ] 0 --.-KB/s \rtest.en-any.en.1 100%[===================>] 271.28K --.-KB/s in 0.02s \n", "\n", "2019-12-31 06:41:57 (16.5 MB/s) - ‘test.en-any.en.1’ saved [277791/277791]\n", "\n", "--2019-12-31 06:41:59-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-urh.en\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 201504 (197K) [text/plain]\n", "Saving to: ‘test.en-urh.en’\n", "\n", "test.en-urh.en 100%[===================>] 196.78K --.-KB/s in 0.01s \n", "\n", "2019-12-31 06:41:59 (13.8 MB/s) - ‘test.en-urh.en’ saved [201504/201504]\n", "\n", "--2019-12-31 06:42:02-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-urh.urh\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 236859 (231K) [text/plain]\n", "Saving to: ‘test.en-urh.urh’\n", "\n", "test.en-urh.urh 100%[===================>] 231.31K --.-KB/s in 0.02s \n", "\n", "2019-12-31 06:42:03 (14.5 MB/s) - ‘test.en-urh.urh’ saved [236859/236859]\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "NqDG-CI28y2L", "colab_type": "code", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "d11c61cb-d165-4fcd-cb20-92015d712ce5" }, "source": [ "# Read the test data to filter from train and dev splits.\n", "# Store english portion in set for quick filtering checks.\n", "en_test_sents = set()\n", "filter_test_sents = \"test.en-any.en\"\n", "j = 0\n", "with open(filter_test_sents) as f:\n", " for line in f:\n", " en_test_sents.add(line.strip())\n", " j += 1\n", "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))" ], "execution_count": 32, "outputs": [ { "output_type": "stream", "text": [ "Loaded 3571 global test sentences to filter from the training/dev data.\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "3CNdwLBCfSIl", "colab": { "base_uri": "https://localhost:8080/", "height": 159 }, "outputId": "077bb76e-99f2-49e4-f539-1cd7a206eefa" }, "source": [ "import pandas as pd\n", "\n", "# TMX file to dataframe\n", "source_file = 'jw300.' + source_language\n", "target_file = 'jw300.' + target_language\n", "\n", "source = []\n", "target = []\n", "skip_lines = [] # Collect the line numbers of the source portion to skip the same lines for the target portion.\n", "with open(source_file) as f:\n", " for i, line in enumerate(f):\n", " # Skip sentences that are contained in the test set.\n", " if line.strip() not in en_test_sents:\n", " source.append(line.strip())\n", " else:\n", " skip_lines.append(i) \n", "with open(target_file) as f:\n", " for j, line in enumerate(f):\n", " # Only add to corpus if corresponding source was not skipped.\n", " if j not in skip_lines:\n", " target.append(line.strip())\n", " \n", "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n", " \n", "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n", "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n", "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n", "df.head(3)" ], "execution_count": 33, "outputs": [ { "output_type": "stream", "text": [ "Loaded data and skipped 4050/32709 lines since contained in test set.\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/html": [ "
\n", " | source_sentence | \n", "target_sentence | \n", "
---|---|---|
0 | \n", "Why It Pays to Be Honest 6 | \n", "Erere Herọ Ra Vwọ Dia Ohwo rẹ Uyota 5 | \n", "
1 | \n", "The Bible Changes Lives | \n", "7 Ovwan “ Jẹn Ẹguọnọ rẹ Iniọvo na Dje Ebuoebuo... | \n", "
2 | \n", "Give Me Just One Year of Peace and Happiness 8... | \n", "12 Jẹ ‘ Ẹse rẹ Ọghẹnẹ rẹ Unu se Gbe - e na , ’... | \n", "