en-ar/main/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139
    },
    "colab_type": "code",
    "id": "oGRmDELn7Az0",
    "outputId": "37f707b9-45b6-46e6-f545-bc00b31f998d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.6/dist-packages (0.4)\n",
      "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
      "\n",
      "Enter your authorization code:\n",
      "··········\n",
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "! pip install googledrivedownloader\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"ar\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# This will save it to a folder in our gdrive instead!\n",
    "!mkdir -p \"masakhane/$src-$tgt-$tag\"\n",
    "os.environ[\"gdrive_path\"] = \"masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kBSgJHEw7Nvx",
    "outputId": "9c544c51-4c87-40e2-f063-25cccd806739"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "masakhane/en-ar-baseline\n"
     ]
    }
   ],
   "source": [
    "!echo $gdrive_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g_9AAHS10I_z"
   },
   "source": [
    "## Used Datasets\n",
    " * JW300\n",
    " * TED TALK\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KGhNlkx90I_2"
   },
   "source": [
    "### JW300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "id": "gA75Fs9ys8Y9",
    "outputId": "42c6fae2-12a3-44b0-be3a-cdcb41d0be43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting opustools-pkg\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
      "\r",
      "\u001b[K     |████                            | 10kB 31.9MB/s eta 0:00:01\r",
      "\u001b[K     |████████                        | 20kB 5.6MB/s eta 0:00:01\r",
      "\u001b[K     |████████████▏                   | 30kB 7.9MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████▏               | 40kB 5.3MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▎           | 51kB 5.7MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▎       | 61kB 6.8MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 71kB 7.6MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 81kB 4.8MB/s \n",
      "\u001b[?25hInstalling collected packages: opustools-pkg\n",
      "Successfully installed opustools-pkg-0.0.52\n"
     ]
    }
   ],
   "source": [
    "# Install opus-tools\n",
    "! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "xq-tDZVks7ZD",
    "outputId": "8880aa17-243a-4a67-a33e-aa9286805895"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/ar-en.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "   5 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/ar-en.xml.gz\n",
      "  55 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/ar.zip\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "\n",
      " 323 MB Total size\n",
      "./JW300_latest_xml_ar-en.xml.gz ... 100% of 5 MB\n",
      "./JW300_latest_xml_ar.zip ... 100% of 55 MB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "gzip: JW300_latest_xml_en-ar.xml.gz: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "# Downloading our corpus\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "colab_type": "code",
    "id": "n48GDRnP8y2G",
    "outputId": "d4df7604-1101-409c-9c4f-803fb4436b46"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-14 18:09:35--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en’\n",
      "\n",
      "\r",
      "test.en-any.en        0%[                    ]       0  --.-KB/s               \r",
      "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.02s   \n",
      "\n",
      "2020-02-14 18:09:36 (15.2 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
      "\n",
      "--2020-02-14 18:09:38--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ar.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 404 Not Found\n",
      "2020-02-14 18:09:38 ERROR 404: Not Found.\n",
      "\n",
      "mv: cannot stat 'test.en-ar.en': No such file or directory\n",
      "--2020-02-14 18:09:44--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ar.ar\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 404 Not Found\n",
      "2020-02-14 18:09:44 ERROR 404: Not Found.\n",
      "\n",
      "mv: cannot stat 'test.en-ar.ar': No such file or directory\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NqDG-CI28y2L",
    "outputId": "7e008cf2-97fa-4332-d95a-2c156b7c2bdb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "3CNdwLBCfSIl",
    "outputId": "b7b73d72-fef9-4713-de2c-db092b14d927"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 3135/547133 lines since contained in test set.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "test_src = \"test.en\"\n",
    "test_trg = \"test.ar\"\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f, open(test_src, 'a') as t:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)\n",
    "            print(line.strip(), file= t)             \n",
    "with open(target_file) as f, open(test_trg, 'a') as t:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "        else:\n",
    "            print(line.strip(), file= t)\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "# df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "copy_df = df "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xk2-OGk81bpM"
   },
   "source": [
    "## Next cells describe the inital pre-processing that was performed on the text\n",
    "- we remove the punctuation from Arabic and English texts.\n",
    "- We remove arabic diacritics. \n",
    "- We then normalize the text. \n",
    "- we finally remove repeating characters. \n",
    "- The above steps were performed with regular experessions and simple Python string processing. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MPmkQIcjdk3u"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "import sys\n",
    "import argparse\n",
    "\n",
    "arabic_punctuations = '''`÷×؛<>_()*&^%][ـ،/:\"؟.,'{}~¦+|!”…“–ـ'''\n",
    "english_punctuations = string.punctuation\n",
    "punctuations_list = arabic_punctuations + english_punctuations\n",
    "\n",
    "arabic_diacritics = re.compile(\"\"\"\n",
    "                             ّ    | # Tashdid\n",
    "                             َ    | # Fatha\n",
    "                             ً    | # Tanwin Fath\n",
    "                             ُ    | # Damma\n",
    "                             ٌ    | # Tanwin Damm\n",
    "                             ِ    | # Kasra\n",
    "                             ٍ    | # Tanwin Kasr\n",
    "                             ْ    | # Sukun\n",
    "                             ـ     # Tatwil/Kashida\n",
    "                         \"\"\", re.VERBOSE)\n",
    "\n",
    "\n",
    "def normalize_arabic(text):\n",
    "    text = re.sub(\"[إأآا]\", \"ا\", text)\n",
    "    text = re.sub(\"ى\", \"ي\", text)\n",
    "    text = re.sub(\"ؤ\", \"ء\", text)\n",
    "    text = re.sub(\"ئ\", \"ء\", text)\n",
    "    text = re.sub(\"ة\", \"ه\", text)\n",
    "    text = re.sub(\"گ\", \"ك\", text)\n",
    "    return text\n",
    "\n",
    "\n",
    "def remove_diacritics(text):\n",
    "    text = re.sub(arabic_diacritics, '', text)\n",
    "    return text\n",
    "\n",
    "\n",
    "def remove_punctuations(text):\n",
    "    translator = str.maketrans('', '', punctuations_list)\n",
    "    return text.translate(translator)\n",
    "\n",
    "\n",
    "def remove_repeating_char(text):\n",
    "    return re.sub(r'(.)\\1+', r'\\1', text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sXhU7ScgdpF9"
   },
   "outputs": [],
   "source": [
    "def process(text):\n",
    "  text = normalize_arabic(text)\n",
    "  text = remove_diacritics(text)\n",
    "  text = remove_punctuations(text)\n",
    "  text = remove_repeating_char(text)\n",
    "\n",
    "  return text "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "9VbLeDuF0JAY",
    "outputId": "5d8b284c-29f9-45c0-b794-1358832b906b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                          source_sentence                                    target_sentence\n",
      "543989                                                     هناك اسباب عديده ‏ لٰكن اهمها ان ٱلكتاب ٱلمقدس...\n",
      "543990  26 . Why are astrology and fortune - telling n...                 لماذا نلتزم بٱلدعوات ٱلتي نقبلها ‏\n",
      "543991  There are various reasons , but the prime reas...   اذا قبلنا دعوه ‏ يجب ان نلتزم بكلمتنا اذا امكن ‏\n",
      "543992  If we have accepted an invitation , we should ...  ‏ مز ١٥ ‏ ٤‏ ‏ لا يجب ان نعتذر عنها لاسباب تافه ‏\n",
      "543993  The host likely has worked hard to prepare the...  والا يضيع تعب مضيفنا وتحضيراته ‏ —‏ ب١٨ ‏ ٣ ص ...\n",
      "543994  18 . Appointed men can learn what lessons from...            ماذا يتعلم ٱلشيوخ وٱلخدام من تيموثاوس ‏\n",
      "543995  Timothy truly cared for people and put spiritu...  يتعلم ٱلشيوخ وٱلخدام ٱلمساعدون ٱلكثير من مثال ...\n",
      "543996  He worked hard in sacred service and applied t...  فقد ٱهتم بغيره ٱهتماما حقيقيا واعطي ٱلاولويه ل...\n",
      "543997  He kept training himself and relied on Jehovah...    كما انه ٱجتهد في ٱلخدمه ٱلمقدسه وطبق ما تعلمه ‏\n",
      "                                          source_sentence                                    target_sentence\n",
      "543989                                                     هناك اسباب عديده ‏ لٰكن اهمها ان ٱلكتاب ٱلمقدس...\n",
      "543990  26 . Why are astrology and fortune - telling n...                 لماذا نلتزم بٱلدعوات ٱلتي نقبلها ‏\n",
      "543991  There are various reasons , but the prime reas...   اذا قبلنا دعوه ‏ يجب ان نلتزم بكلمتنا اذا امكن ‏\n",
      "543992  If we have accepted an invitation , we should ...  ‏ مز ١٥ ‏ ٤‏ ‏ لا يجب ان نعتذر عنها لاسباب تافه ‏\n",
      "543993  The host likely has worked hard to prepare the...  والا يضيع تعب مضيفنا وتحضيراته ‏ —‏ ب١٨ ‏ ٣ ص ...\n",
      "543994  18 . Appointed men can learn what lessons from...            ماذا يتعلم ٱلشيوخ وٱلخدام من تيموثاوس ‏\n",
      "543995  Timothy truly cared for people and put spiritu...  يتعلم ٱلشيوخ وٱلخدام ٱلمساعدون ٱلكثير من مثال ...\n",
      "543996  He worked hard in sacred service and applied t...  فقد ٱهتم بغيره ٱهتماما حقيقيا واعطي ٱلاولويه ل...\n",
      "543997  He kept training himself and relied on Jehovah...    كما انه ٱجتهد في ٱلخدمه ٱلمقدسه وطبق ما تعلمه ‏\n"
     ]
    }
   ],
   "source": [
    "for index, row in copy_df.iterrows():\n",
    "  s = row['target_sentence']\n",
    "  row['target_sentence'] = process(s)\n",
    "\n",
    "print(copy_df[-10:-1])\n",
    "print(df[-10:-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "QCfM74A2f70J",
    "outputId": "9d3813f0-0c87-423e-ec8f-a647f0dc1d0b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "هناك اسباب عديده ‏ لٰكن اهم\n"
     ]
    }
   ],
   "source": [
    "text = process(\"هُنَاكَ أَسْبَابٌ عَدِيدَةٌ ،‏ لٰكِنَّ أَهَمَّ\")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MYGo1eg0Bisv"
   },
   "outputs": [],
   "source": [
    "copy_df.to_csv(\"edited_arabic-english_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vRhANMoA0JAd"
   },
   "source": [
    "### TED Talk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_vWnvgWH1bp6"
   },
   "source": [
    "### TED Talk data intial preprocessing \n",
    "The data was cleaned using MS-Excel because it was more convinent to use. The text didn't contain diacritics but it contained spaces, and NULL characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "G0UNOkKO0JAf",
    "outputId": "ee8083c0-6745-457d-bfa1-64eb1227dda2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: ezodf in /usr/local/lib/python3.6/dist-packages (0.3.2)\n"
     ]
    }
   ],
   "source": [
    "#IN Order to get the dataframe out of the .ods file please intall ezodf\n",
    "! pip install ezodf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WHp0PS7NBRTT"
   },
   "outputs": [],
   "source": [
    "url = 'https://drive.google.com/uc?export=download&id=1oNU9PA47cEc6HfuD1gza44502GNQCLew'\n",
    "\n",
    "import requests\n",
    "r = requests.get(url) \n",
    "with open(\"processed_ted_talk_train_data.ods\", \"wb\") as code:\n",
    "    code.write(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k_S93Qva0JAj"
   },
   "outputs": [],
   "source": [
    "import ezodf\n",
    "def read_ods(filename, sheet_no=0, header=0):\n",
    "    tab = ezodf.opendoc(filename=filename).sheets[sheet_no]\n",
    "    return pd.DataFrame({col[header].value:[x.value for x in col[header+1:]]\n",
    "                         for col in tab.columns()})\n",
    "\n",
    "df_ted = read_ods(\"processed_ted_talk_train_data.ods\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "JfFja8Sx2U2W",
    "outputId": "1b7632ff-dc5e-4baa-9019-4c829ab7c680"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>ar</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amongst all the troubling deficits we struggle...</td>\n",
       "      <td>من ضمن جميع المثبطات المقلقة التي نعاني منها ا...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>We who are diplomats , we are trained to deal ...</td>\n",
       "      <td>نحن الدبلوماسيين تدربنا على التعامل مع الصراعا...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>And I can tell you , our agenda is full .</td>\n",
       "      <td>وبامكاني القول ان جدول اعمالنا ممتلئ</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>There is trade , there is disarmament , there ...</td>\n",
       "      <td>هناك التجارة , هناك نزع السلاح , وهناك علاقات ...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>But the picture is changing , and we are seein...</td>\n",
       "      <td>لكن الصورة تتغير ونحن نرى ان هناك لاعبين رئيسي...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  en  ...   NaN\n",
       "0  Amongst all the troubling deficits we struggle...  ...  None\n",
       "1  We who are diplomats , we are trained to deal ...  ...  None\n",
       "2          And I can tell you , our agenda is full .  ...  None\n",
       "3  There is trade , there is disarmament , there ...  ...  None\n",
       "4  But the picture is changing , and we are seein...  ...  None\n",
       "\n",
       "[5 rows x 3 columns]"
      ]
     },
     "execution_count": 76,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ted.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "ZXantb8m0JAo",
    "outputId": "f9c56c7f-3b79-4668-bb9f-85de181284eb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amongst all the troubling deficits we struggle...</td>\n",
       "      <td>من ضمن جميع المثبطات المقلقة التي نعاني منها ا...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>We who are diplomats , we are trained to deal ...</td>\n",
       "      <td>نحن الدبلوماسيين تدربنا على التعامل مع الصراعا...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>And I can tell you , our agenda is full .</td>\n",
       "      <td>وبامكاني القول ان جدول اعمالنا ممتلئ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>There is trade , there is disarmament , there ...</td>\n",
       "      <td>هناك التجارة , هناك نزع السلاح , وهناك علاقات ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>But the picture is changing , and we are seein...</td>\n",
       "      <td>لكن الصورة تتغير ونحن نرى ان هناك لاعبين رئيسي...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence                                    target_sentence\n",
       "0  Amongst all the troubling deficits we struggle...  من ضمن جميع المثبطات المقلقة التي نعاني منها ا...\n",
       "1  We who are diplomats , we are trained to deal ...  نحن الدبلوماسيين تدربنا على التعامل مع الصراعا...\n",
       "2          And I can tell you , our agenda is full .               وبامكاني القول ان جدول اعمالنا ممتلئ\n",
       "3  There is trade , there is disarmament , there ...  هناك التجارة , هناك نزع السلاح , وهناك علاقات ...\n",
       "4  But the picture is changing , and we are seein...  لكن الصورة تتغير ونحن نرى ان هناك لاعبين رئيسي..."
      ]
     },
     "execution_count": 78,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## remove the Nan Column + display head\n",
    "df_ted = df_ted[[\"en\",'ar']]\n",
    "df_ted.head()\n",
    "df_ted.columns = [\"source_sentence\", \"target_sentence\"]\n",
    "copy_ted = df_ted \n",
    "\n",
    "copy_ted.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CHq8EjYv0JAt"
   },
   "source": [
    "## Merging Tedtalk and JW300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MikpwH_I0JAv"
   },
   "outputs": [],
   "source": [
    "df = copy_df.append(copy_ted, ignore_index=True)\n",
    "df.shape\n",
    "df.to_csv(\"full_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "bwMBTYxzTkac",
    "outputId": "9eaffe51-30d5-4abc-84d8-7058af04a21b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "repeated values describtion \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count     765188\n",
       "unique         2\n",
       "top        False\n",
       "freq      732429\n",
       "dtype: object"
      ]
     },
     "execution_count": 108,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_dups = df.duplicated(subset=None, keep='first')\n",
    "print(\"repeated values describtion \")\n",
    "find_dups.describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ratio of non repateated entries in the two datasets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "jwkN-BaFUMkg",
    "outputId": "76f7da45-8aae-437c-b855-688971a6bff0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio of un-repeated values =  0.9571882988232957\n"
     ]
    }
   ],
   "source": [
    "\n",
    "output = find_dups.describe()\n",
    "ratio = output.freq/ len(find_dups)\n",
    "print(\"ratio of un-repeated values = \", ratio)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M_2ouEOH1_1q"
   },
   "outputs": [],
   "source": [
    "# drop duplicate translations\n",
    "df = df.drop_duplicates()\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "# df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "# df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df = df.sample(frac=1, random_state=seed).reset_index(drop=True)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SdJ8ONao1bq6"
   },
   "outputs": [],
   "source": [
    "#GET THE TEST DATA FROM THE TOTAL DATAFRAME\n",
    "df_test = df.loc[:3751,:]\n",
    "df_train = df.loc[3751:,:]\n",
    "print(df_test.shape, df_train.shape)\n",
    "en_test_sents = df_test.source_sentence.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z_1BwAApEtMk"
   },
   "outputs": [],
   "source": [
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_train.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan\n",
    "\n",
    "# NOTE - This might run slow depending on the size of your training set. We are\n",
    "# printing some information to help you track how long it would take. \n",
    "scores = []\n",
    "start_time = time.time()\n",
    "for idx, row in df_train.iterrows():\n",
    "  scores.append(fuzzfilter(row['source_sentence'], en_test_sents, 5))\n",
    "  if idx % 1000 == 0:\n",
    "    hours, rem = divmod(time.time() - start_time, 3600)\n",
    "    minutes, seconds = divmod(rem, 60)\n",
    "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_train))))\n",
    "\n",
    "# Filter out \"almost overlapping samples\"\n",
    "df_train['scores'] = scores\n",
    "df_train = df_train[df_train['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hxxBOCA-xXhy"
   },
   "outputs": [],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_train[\"source_sentence\"] = df_train[\"source_sentence\"].str.lower()\n",
    "    df_train[\"target_sentence\"] = df_train[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_train.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_train.drop(df_train.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    if type(row[\"source_sentence\"]) == str:\n",
    "        print(row[\"source_sentence\"]+\"\\n\", file=src_file)\n",
    "    else:\n",
    "        continue\n",
    "    if type(row[\"target_sentence\"]) == str:\n",
    "        print(row[\"target_sentence\"]+\"\\n\", file=trg_file)\n",
    "    else:\n",
    "        continue\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    if type(row[\"source_sentence\"]) == str:\n",
    "        print(row[\"source_sentence\"]+\"\\n\",file=src_file)\n",
    "    else:\n",
    "        continue\n",
    "    \n",
    "    if type(row[\"target_sentence\"]) == str:\n",
    "        print(row[\"target_sentence\"]+\"\\n\",file=trg_file)\n",
    "    else:\n",
    "        continue\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iBRMm4kMxZ8L"
   },
   "outputs": [],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H-TyjtmXB1mL"
   },
   "outputs": [],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Arabic Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IlMitUHR8Qy-"
   },
   "outputs": [],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIs1lY2hxMsl"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (source_language, target_language)\n",
    "gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 1000\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3570\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 30                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 60800          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_transformer\"\n",
    "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
    "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pIifxE3Qzuvs"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6ZBPFwT94WpI"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n94wlrCjVc17"
   },
   "outputs": [],
   "source": [
    "# Output our validation accuracy\n",
    "! tail \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "66WhRE9lIhoD"
   },
   "outputs": [],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "notebook.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}