{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## English - Tigrinya\n",
    "\n",
    "### Data\n",
    "A mix of corpora is used: \n",
    "- JW300\n",
    "- Tatoeba\n",
    "- [Parallel Corpora for Ethiopian Languages](https://github.com/AAUThematic4LT/Parallel-Corpora-for-Ethiopian-Languages)\n",
    "\n",
    "Test set is retrieved from masakhane global JW300 test set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 124
    },
    "colab_type": "code",
    "id": "oGRmDELn7Az0",
    "outputId": "4cf88906-25db-48d8-db25-6395cc4dc5d3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
      "\n",
      "Enter your authorization code:\n",
      "··········\n",
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"ti\" \n",
    "lc = True  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"tigmix\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# This will save it to a folder in our gdrive instead!\n",
    "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
    "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kBSgJHEw7Nvx",
    "outputId": "171a2695-9c9e-4bf6-a2bf-1ebd440bce6d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/masakhane/en-ti-tigmix\n"
     ]
    }
   ],
   "source": [
    "!echo $gdrive_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "gA75Fs9ys8Y9",
    "outputId": "af082e56-690f-4395-8ef4-31740fd0fd37"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: opustools-pkg in /usr/local/lib/python3.6/dist-packages (0.0.52)\n"
     ]
    }
   ],
   "source": [
    "# Install opus-tools\n",
    "! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 697
    },
    "colab_type": "code",
    "id": "xq-tDZVks7ZD",
    "outputId": "161889c4-4f43-4f38-e2dc-53930bd2b59e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-ti.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "   4 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-ti.xml.gz\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "  40 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/ti.zip\n",
      "\n",
      " 307 MB Total size\n",
      "./JW300_latest_xml_en-ti.xml.gz ... 100% of 4 MB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "./JW300_latest_xml_ti.zip ... 100% of 40 MB\n",
      "--2020-01-21 10:47:26--  https://object.pouta.csc.fi/OPUS-Tatoeba/v20190709/moses/en-ti.txt.zip\n",
      "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.18, 86.50.254.19\n",
      "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 9408 (9.2K) [application/zip]\n",
      "Saving to: ‘en-ti.txt.zip’\n",
      "\n",
      "en-ti.txt.zip       100%[===================>]   9.19K  --.-KB/s    in 0s      \n",
      "\n",
      "2020-01-21 10:47:27 (319 MB/s) - ‘en-ti.txt.zip’ saved [9408/9408]\n",
      "\n",
      "Archive:  en-ti.txt.zip\n",
      "  inflating: README                  \n",
      "  inflating: LICENSE                 \n",
      "replace Tatoeba.en-ti.en? [y]es, [n]o, [A]ll, [N]one, [r]ename: y\n",
      "  inflating: Tatoeba.en-ti.en        \n",
      "replace Tatoeba.en-ti.ti? [y]es, [n]o, [A]ll, [N]one, [r]ename: y\n",
      "  inflating: Tatoeba.en-ti.ti        \n",
      "  inflating: Tatoeba.en-ti.xml       \n",
      "Total lines from Ethiopian corpus:\n",
      "36025 ethiopian.ti\n",
      "36025 ethiopian.en\n",
      "cat: tigmix.en: input file is output file\n",
      "cat: tigmix.ti: input file is output file\n",
      "Total lines Tigmix:\n",
      "435546 tigmix.ti\n",
      "435546 tigmix.en\n"
     ]
    }
   ],
   "source": [
    "# Downloadi and extract JW300\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz\n",
    "\n",
    "# Downloading other corpora\n",
    "# OPUS Tatoeba\n",
    "! wget https://object.pouta.csc.fi/OPUS-Tatoeba/v20190709/moses/en-ti.txt.zip\n",
    "! unzip en-ti.txt.zip\n",
    "! rm *.zip LICENSE README *.xml\n",
    "\n",
    "# Ethiopian languages corpus\n",
    "#! git clone https://github.com/AAUThematic4LT/Parallel-Corpora-for-Ethiopian-Languages.git\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Legal/tig_eng/other.eg > ethiopian.en\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Legal/tig_eng/other.tg > ethiopian.ti\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Legal/tig_eng/p_eng_et.txt >> ethiopian.en\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Legal/tig_eng/p_tig_et.txt >> ethiopian.ti\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Spritual/jw_bible/tig_eng/p_eng_et.txt >> ethiopian.en\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Spritual/jw_bible/tig_eng/p_tig_et.txt >> ethiopian.ti\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Spritual/jw_daily/tig_eng/p_eng_et >> ethiopian.en\n",
    "! cat Parallel-Corpora-for-Ethiopian-Languages/Exp\\ I-English\\ to\\ Local\\ Lang/Spritual/jw_daily/tig_eng/p_tig_et >> ethiopian.ti\n",
    "\n",
    "print(\"Total lines from Ethiopian corpus:\")\n",
    "! wc -l ethiopian.ti\n",
    "! wc -l ethiopian.en\n",
    "\n",
    "# Merge all corpora into one dataset: Tigmix\n",
    "! cat *.en > tigmix.en\n",
    "! cat *.ti > tigmix.ti\n",
    "\n",
    "print(\"Total lines Tigmix:\")\n",
    "! wc -l tigmix.ti\n",
    "! wc -l tigmix.en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 610
    },
    "colab_type": "code",
    "id": "n48GDRnP8y2G",
    "outputId": "4506c1b1-f0c1-4188-c017-68c23a249570"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-01-21 10:49:40--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en’\n",
      "\n",
      "\r",
      "test.en-any.en        0%[                    ]       0  --.-KB/s               \r",
      "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2020-01-21 10:49:40 (6.62 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
      "\n",
      "--2020-01-21 10:49:41--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ti.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 199625 (195K) [text/plain]\n",
      "Saving to: ‘test.en-ti.en’\n",
      "\n",
      "test.en-ti.en       100%[===================>] 194.95K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2020-01-21 10:49:42 (5.35 MB/s) - ‘test.en-ti.en’ saved [199625/199625]\n",
      "\n",
      "--2020-01-21 10:49:44--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ti.ti\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 311059 (304K) [text/plain]\n",
      "Saving to: ‘test.en-ti.ti’\n",
      "\n",
      "test.en-ti.ti       100%[===================>] 303.77K  --.-KB/s    in 0.05s   \n",
      "\n",
      "2020-01-21 10:49:44 (5.92 MB/s) - ‘test.en-ti.ti’ saved [311059/311059]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NqDG-CI28y2L",
    "outputId": "3e2177ae-2a94-4b9a-8ac5-dcdad6e8c844"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 196
    },
    "colab_type": "code",
    "id": "3CNdwLBCfSIl",
    "outputId": "b0d30018-76a8-42f5-f8d3-19ad1c9df3aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 31869/435545 lines either empty or contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Health care for asylum seekers and refugees in...</td>\n",
       "      <td>ሓልዮት ጥዕና ንደለይቲ ዑቕባን ስደተኛታትን ኣብ ስኮትላንድ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>In Scotland , most health care is provided by ...</td>\n",
       "      <td>መብዛሕትኡ ክንክን ጥዕና ኣብ ስኮትላንድ ብሃገራዊ ኣገልግሎት ጥዕና ( ሃ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Everyone who lives legally in Scotland has a r...</td>\n",
       "      <td>ሕጋዊያን ነበርቲ ስኮትላንድ ፤ ዜግነቶም ብዘየገድስ መሰል ናይ ሃ.ኣ.ጥ....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence                                    target_sentence\n",
       "0  Health care for asylum seekers and refugees in...              ሓልዮት ጥዕና ንደለይቲ ዑቕባን ስደተኛታትን ኣብ ስኮትላንድ\n",
       "1  In Scotland , most health care is provided by ...  መብዛሕትኡ ክንክን ጥዕና ኣብ ስኮትላንድ ብሃገራዊ ኣገልግሎት ጥዕና ( ሃ...\n",
       "2  Everyone who lives legally in Scotland has a r...  ሕጋዊያን ነበርቲ ስኮትላንድ ፤ ዜግነቶም ብዘየገድስ መሰል ናይ ሃ.ኣ.ጥ...."
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'tigmix.' + source_language\n",
    "target_file = 'tigmix.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "no_skip_lines = 0  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f_src, open(target_file) as f_tgt:\n",
    "    for line_src, line_tgt in zip(f_src, f_tgt):\n",
    "        # Skip sentences that are empty and contained in the test set.\n",
    "        if line_src.strip() not in en_test_sents and line_src.strip() and line_tgt.strip():\n",
    "          source.append(line_src.strip())         \n",
    "          target.append(line_tgt.strip())\n",
    "        else:\n",
    "          no_skip_lines += 1\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines either empty or contained in test set.'.format(no_skip_lines, i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 86
    },
    "colab_type": "code",
    "id": "M_2ouEOH1_1q",
    "outputId": "8160258d-26f1-4286-8141-47c02d64fb8d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current corpus size: 403677\n",
      "After drop duplicates: 384332\n",
      "After drop non-geez on target: 381449\n",
      "After drop geez on source: 381350\n"
     ]
    }
   ],
   "source": [
    "print(\"Current corpus size: %i\"%(len(df)))\n",
    "\n",
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "print(\"After drop duplicates: %i\"%len(df_pp))\n",
    "\n",
    "#drop non-geez lines from target and geez lines in source\n",
    "df_pp = df_pp[df_pp.target_sentence.str.contains('[\\u1200-\\u137f|\\u1380-\\u1394|\\u2d80-\\u2ddf|\\uab00-\\uab2f]',case=False)]\n",
    "print(\"After drop non-geez on target: %i\"%len(df_pp))\n",
    "df_pp = df_pp[-df_pp.source_sentence.str.contains('[\\u1200-\\u137f|\\u1380-\\u1394|\\u2d80-\\u2ddf|\\uab00-\\uab2f]',case=False)]\n",
    "print(\"After drop geez on source: %i\"%len(df_pp))\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "# df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "# df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "# print(\"After drop conflicts: %i\"%len(df_pp))\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "Z_1BwAApEtMk",
    "outputId": "dcb24c32-63f7-4c6d-ff07-9381de496aea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fuzzywuzzy\n",
      "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
      "Installing collected packages: fuzzywuzzy\n",
      "Successfully installed fuzzywuzzy-0.17.0\n",
      "Collecting python-Levenshtein\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 2.1MB/s \n",
      "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (42.0.2)\n",
      "Building wheels for collected packages: python-Levenshtein\n",
      "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144672 sha256=9797a4dd64722f4a7f909ddc355dd961c72cf3a5ade89720476211003ba9713a\n",
      "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
      "Successfully built python-Levenshtein\n",
      "Installing collected packages: python-Levenshtein\n",
      "Successfully installed python-Levenshtein-0.12.0\n",
      "00:00:00.11 0.00 percent complete\n",
      "00:00:18.99 0.26 percent complete\n",
      "00:00:39.53 0.52 percent complete\n",
      "00:00:59.44 0.79 percent complete\n",
      "00:01:18.94 1.05 percent complete\n",
      "00:01:39.10 1.31 percent complete\n",
      "00:01:58.76 1.57 percent complete\n",
      "00:02:18.27 1.84 percent complete\n",
      "00:02:38.05 2.10 percent complete\n",
      "00:02:57.15 2.36 percent complete\n",
      "00:03:16.39 2.62 percent complete\n",
      "00:03:37.22 2.88 percent complete\n",
      "00:03:57.15 3.15 percent complete\n",
      "00:04:16.54 3.41 percent complete\n",
      "00:04:37.01 3.67 percent complete\n",
      "00:04:56.55 3.93 percent complete\n",
      "00:05:15.93 4.20 percent complete\n",
      "00:05:35.88 4.46 percent complete\n",
      "00:05:55.40 4.72 percent complete\n",
      "00:06:15.30 4.98 percent complete\n",
      "00:06:34.45 5.24 percent complete\n",
      "00:06:53.79 5.51 percent complete\n",
      "00:07:12.97 5.77 percent complete\n",
      "00:07:32.99 6.03 percent complete\n",
      "00:07:52.84 6.29 percent complete\n",
      "00:08:12.10 6.56 percent complete\n",
      "00:08:32.22 6.82 percent complete\n",
      "00:08:52.03 7.08 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:09:12.12 7.34 percent complete\n",
      "00:09:31.16 7.60 percent complete\n",
      "00:09:51.14 7.87 percent complete\n",
      "00:10:10.42 8.13 percent complete\n",
      "00:10:30.60 8.39 percent complete\n",
      "00:10:49.70 8.65 percent complete\n",
      "00:11:08.90 8.92 percent complete\n",
      "00:11:28.76 9.18 percent complete\n",
      "00:11:48.24 9.44 percent complete\n",
      "00:12:07.35 9.70 percent complete\n",
      "00:12:26.81 9.96 percent complete\n",
      "00:12:46.85 10.23 percent complete\n",
      "00:13:05.54 10.49 percent complete\n",
      "00:13:24.39 10.75 percent complete\n",
      "00:13:43.78 11.01 percent complete\n",
      "00:14:03.56 11.28 percent complete\n",
      "00:14:23.03 11.54 percent complete\n",
      "00:14:42.33 11.80 percent complete\n",
      "00:15:01.12 12.06 percent complete\n",
      "00:15:20.67 12.32 percent complete\n",
      "00:15:39.32 12.59 percent complete\n",
      "00:15:58.35 12.85 percent complete\n",
      "00:16:17.89 13.11 percent complete\n",
      "00:16:37.35 13.37 percent complete\n",
      "00:16:57.24 13.64 percent complete\n",
      "00:17:16.98 13.90 percent complete\n",
      "00:17:35.67 14.16 percent complete\n",
      "00:17:55.11 14.42 percent complete\n",
      "00:18:14.52 14.68 percent complete\n",
      "00:18:33.86 14.95 percent complete\n",
      "00:18:54.07 15.21 percent complete\n",
      "00:19:14.08 15.47 percent complete\n",
      "00:19:34.73 15.73 percent complete\n",
      "00:19:54.02 16.00 percent complete\n",
      "00:20:13.85 16.26 percent complete\n",
      "00:20:33.48 16.52 percent complete\n",
      "00:20:53.04 16.78 percent complete\n",
      "00:21:12.39 17.04 percent complete\n",
      "00:21:31.52 17.31 percent complete\n",
      "00:21:51.04 17.57 percent complete\n",
      "00:22:10.54 17.83 percent complete\n",
      "00:22:30.22 18.09 percent complete\n",
      "00:22:49.60 18.36 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:23:08.66 18.62 percent complete\n",
      "00:23:28.63 18.88 percent complete\n",
      "00:23:47.25 19.14 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:24:06.11 19.40 percent complete\n",
      "00:24:24.76 19.67 percent complete\n",
      "00:24:45.00 19.93 percent complete\n",
      "00:25:04.04 20.19 percent complete\n",
      "00:25:22.89 20.45 percent complete\n",
      "00:25:42.27 20.72 percent complete\n",
      "00:26:01.41 20.98 percent complete\n",
      "00:26:20.61 21.24 percent complete\n",
      "00:26:39.83 21.50 percent complete\n",
      "00:26:59.22 21.76 percent complete\n",
      "00:27:18.67 22.03 percent complete\n",
      "00:27:38.12 22.29 percent complete\n",
      "00:27:57.65 22.55 percent complete\n",
      "00:28:16.51 22.81 percent complete\n",
      "00:28:36.11 23.08 percent complete\n",
      "00:28:55.73 23.34 percent complete\n",
      "00:29:14.86 23.60 percent complete\n",
      "00:29:34.88 23.86 percent complete\n",
      "00:29:54.75 24.12 percent complete\n",
      "00:30:13.80 24.39 percent complete\n",
      "00:30:33.49 24.65 percent complete\n",
      "00:30:52.22 24.91 percent complete\n",
      "00:31:11.17 25.17 percent complete\n",
      "00:31:31.01 25.44 percent complete\n",
      "00:31:49.62 25.70 percent complete\n",
      "00:32:07.96 25.96 percent complete\n",
      "00:32:27.45 26.22 percent complete\n",
      "00:32:47.14 26.48 percent complete\n",
      "00:33:06.10 26.75 percent complete\n",
      "00:33:25.68 27.01 percent complete\n",
      "00:33:45.03 27.27 percent complete\n",
      "00:34:04.02 27.53 percent complete\n",
      "00:34:23.27 27.80 percent complete\n",
      "00:34:42.93 28.06 percent complete\n",
      "00:35:02.02 28.32 percent complete\n",
      "00:35:21.86 28.58 percent complete\n",
      "00:35:40.93 28.84 percent complete\n",
      "00:35:59.80 29.11 percent complete\n",
      "00:36:18.65 29.37 percent complete\n",
      "00:36:38.17 29.63 percent complete\n",
      "00:36:57.28 29.89 percent complete\n",
      "00:37:16.69 30.16 percent complete\n",
      "00:37:37.09 30.42 percent complete\n",
      "00:37:55.98 30.68 percent complete\n",
      "00:38:15.18 30.94 percent complete\n",
      "00:38:34.72 31.20 percent complete\n",
      "00:38:54.28 31.47 percent complete\n",
      "00:39:13.41 31.73 percent complete\n",
      "00:39:33.11 31.99 percent complete\n",
      "00:39:52.10 32.25 percent complete\n",
      "00:40:11.29 32.52 percent complete\n",
      "00:40:31.09 32.78 percent complete\n",
      "00:40:49.96 33.04 percent complete\n",
      "00:41:08.54 33.30 percent complete\n",
      "00:41:28.14 33.56 percent complete\n",
      "00:41:47.91 33.83 percent complete\n",
      "00:42:06.94 34.09 percent complete\n",
      "00:42:25.48 34.35 percent complete\n",
      "00:42:44.97 34.61 percent complete\n",
      "00:43:04.15 34.88 percent complete\n",
      "00:43:22.87 35.14 percent complete\n",
      "00:43:42.59 35.40 percent complete\n",
      "00:44:01.41 35.66 percent complete\n",
      "00:44:20.67 35.93 percent complete\n",
      "00:44:40.61 36.19 percent complete\n",
      "00:44:59.56 36.45 percent complete\n",
      "00:45:18.64 36.71 percent complete\n",
      "00:45:38.47 36.97 percent complete\n",
      "00:45:57.49 37.24 percent complete\n",
      "00:46:17.30 37.50 percent complete\n",
      "00:46:36.72 37.76 percent complete\n",
      "00:46:56.14 38.02 percent complete\n",
      "00:47:15.33 38.29 percent complete\n",
      "00:47:34.57 38.55 percent complete\n",
      "00:47:54.39 38.81 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:48:13.98 39.07 percent complete\n",
      "00:48:32.86 39.33 percent complete\n",
      "00:48:51.66 39.60 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:49:10.35 39.86 percent complete\n",
      "00:49:29.86 40.12 percent complete\n",
      "00:49:48.49 40.38 percent complete\n",
      "00:50:07.68 40.65 percent complete\n",
      "00:50:26.37 40.91 percent complete\n",
      "00:50:46.08 41.17 percent complete\n",
      "00:51:04.87 41.43 percent complete\n",
      "00:51:23.42 41.69 percent complete\n",
      "00:51:42.23 41.96 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:52:00.30 42.22 percent complete\n",
      "00:52:18.88 42.48 percent complete\n",
      "00:52:37.73 42.74 percent complete\n",
      "00:52:56.90 43.01 percent complete\n",
      "00:53:16.35 43.27 percent complete\n",
      "00:53:35.64 43.53 percent complete\n",
      "00:53:54.44 43.79 percent complete\n",
      "00:54:13.42 44.05 percent complete\n",
      "00:54:32.04 44.32 percent complete\n",
      "00:54:51.35 44.58 percent complete\n",
      "00:55:10.48 44.84 percent complete\n",
      "00:55:29.80 45.10 percent complete\n",
      "00:55:48.64 45.37 percent complete\n",
      "00:56:07.24 45.63 percent complete\n",
      "00:56:26.34 45.89 percent complete\n",
      "00:56:45.82 46.15 percent complete\n",
      "00:57:04.62 46.41 percent complete\n",
      "00:57:23.21 46.68 percent complete\n",
      "00:57:42.58 46.94 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:58:01.64 47.20 percent complete\n",
      "00:58:21.61 47.46 percent complete\n",
      "00:58:40.58 47.73 percent complete\n",
      "00:58:59.28 47.99 percent complete\n",
      "00:59:18.22 48.25 percent complete\n",
      "00:59:37.67 48.51 percent complete\n",
      "00:59:57.05 48.77 percent complete\n",
      "01:00:16.06 49.04 percent complete\n",
      "01:00:35.48 49.30 percent complete\n",
      "01:00:54.57 49.56 percent complete\n",
      "01:01:12.93 49.82 percent complete\n",
      "01:01:32.11 50.09 percent complete\n",
      "01:01:50.87 50.35 percent complete\n",
      "01:02:09.09 50.61 percent complete\n",
      "01:02:28.60 50.87 percent complete\n",
      "01:02:47.35 51.13 percent complete\n",
      "01:03:06.06 51.40 percent complete\n",
      "01:03:25.23 51.66 percent complete\n",
      "01:03:44.86 51.92 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:04:04.01 52.18 percent complete\n",
      "01:04:23.00 52.45 percent complete\n",
      "01:04:42.08 52.71 percent complete\n",
      "01:05:01.18 52.97 percent complete\n",
      "01:05:20.32 53.23 percent complete\n",
      "01:05:39.72 53.49 percent complete\n",
      "01:05:58.89 53.76 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓ ↓ ↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:06:18.15 54.02 percent complete\n",
      "01:06:37.07 54.28 percent complete\n",
      "01:06:55.61 54.54 percent complete\n",
      "01:07:14.71 54.81 percent complete\n",
      "01:07:34.17 55.07 percent complete\n",
      "01:07:53.14 55.33 percent complete\n",
      "01:08:12.30 55.59 percent complete\n",
      "01:08:32.00 55.85 percent complete\n",
      "01:08:51.40 56.12 percent complete\n",
      "01:09:10.40 56.38 percent complete\n",
      "01:09:29.57 56.64 percent complete\n",
      "01:09:49.26 56.90 percent complete\n",
      "01:10:08.45 57.17 percent complete\n",
      "01:10:28.19 57.43 percent complete\n",
      "01:10:47.06 57.69 percent complete\n",
      "01:11:06.01 57.95 percent complete\n",
      "01:11:25.84 58.21 percent complete\n",
      "01:11:45.04 58.48 percent complete\n",
      "01:12:03.76 58.74 percent complete\n",
      "01:12:22.67 59.00 percent complete\n",
      "01:12:42.35 59.26 percent complete\n",
      "01:13:02.09 59.53 percent complete\n",
      "01:13:21.20 59.79 percent complete\n",
      "01:13:41.66 60.05 percent complete\n",
      "01:14:01.21 60.31 percent complete\n",
      "01:14:19.92 60.57 percent complete\n",
      "01:14:39.31 60.84 percent complete\n",
      "01:14:58.07 61.10 percent complete\n",
      "01:15:17.55 61.36 percent complete\n",
      "01:15:37.44 61.62 percent complete\n",
      "01:15:56.41 61.89 percent complete\n",
      "01:16:15.31 62.15 percent complete\n",
      "01:16:35.19 62.41 percent complete\n",
      "01:16:53.45 62.67 percent complete\n",
      "01:17:11.96 62.93 percent complete\n",
      "01:17:31.27 63.20 percent complete\n",
      "01:17:50.58 63.46 percent complete\n",
      "01:18:08.95 63.72 percent complete\n",
      "01:18:28.46 63.98 percent complete\n",
      "01:18:48.07 64.25 percent complete\n",
      "01:19:07.98 64.51 percent complete\n",
      "01:19:26.96 64.77 percent complete\n",
      "01:19:46.91 65.03 percent complete\n",
      "01:20:05.30 65.29 percent complete\n",
      "01:20:24.41 65.56 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:20:43.48 65.82 percent complete\n",
      "01:21:02.63 66.08 percent complete\n",
      "01:21:21.77 66.34 percent complete\n",
      "01:21:42.11 66.61 percent complete\n",
      "01:22:01.09 66.87 percent complete\n",
      "01:22:20.24 67.13 percent complete\n",
      "01:22:39.54 67.39 percent complete\n",
      "01:22:58.07 67.65 percent complete\n",
      "01:23:17.54 67.92 percent complete\n",
      "01:23:36.67 68.18 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:23:55.83 68.44 percent complete\n",
      "01:24:15.81 68.70 percent complete\n",
      "01:24:34.83 68.97 percent complete\n",
      "01:24:54.46 69.23 percent complete\n",
      "01:25:13.82 69.49 percent complete\n",
      "01:25:32.94 69.75 percent complete\n",
      "01:25:52.02 70.01 percent complete\n",
      "01:26:11.45 70.28 percent complete\n",
      "01:26:30.43 70.54 percent complete\n",
      "01:26:49.54 70.80 percent complete\n",
      "01:27:08.28 71.06 percent complete\n",
      "01:27:27.16 71.33 percent complete\n",
      "01:27:46.38 71.59 percent complete\n",
      "01:28:05.35 71.85 percent complete\n",
      "01:28:23.78 72.11 percent complete\n",
      "01:28:43.70 72.37 percent complete\n",
      "01:29:02.88 72.64 percent complete\n",
      "01:29:22.67 72.90 percent complete\n",
      "01:29:42.65 73.16 percent complete\n",
      "01:30:01.59 73.42 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '’—.']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:30:21.34 73.69 percent complete\n",
      "01:30:40.11 73.95 percent complete\n",
      "01:30:59.45 74.21 percent complete\n",
      "01:31:18.74 74.47 percent complete\n",
      "01:31:38.83 74.73 percent complete\n",
      "01:31:58.21 75.00 percent complete\n",
      "01:32:16.49 75.26 percent complete\n",
      "01:32:35.97 75.52 percent complete\n",
      "01:32:54.51 75.78 percent complete\n",
      "01:33:13.40 76.05 percent complete\n",
      "01:33:32.65 76.31 percent complete\n",
      "01:33:51.34 76.57 percent complete\n",
      "01:34:09.97 76.83 percent complete\n",
      "01:34:30.00 77.09 percent complete\n",
      "01:34:48.66 77.36 percent complete\n",
      "01:35:07.42 77.62 percent complete\n",
      "01:35:26.47 77.88 percent complete\n",
      "01:35:45.41 78.14 percent complete\n",
      "01:36:04.79 78.41 percent complete\n",
      "01:36:23.74 78.67 percent complete\n",
      "01:36:43.32 78.93 percent complete\n",
      "01:37:02.92 79.19 percent complete\n",
      "01:37:22.53 79.45 percent complete\n",
      "01:37:41.91 79.72 percent complete\n",
      "01:38:00.18 79.98 percent complete\n",
      "01:38:19.22 80.24 percent complete\n",
      "01:38:38.67 80.50 percent complete\n",
      "01:38:57.91 80.77 percent complete\n",
      "01:39:16.83 81.03 percent complete\n",
      "01:39:36.25 81.29 percent complete\n",
      "01:39:56.05 81.55 percent complete\n",
      "01:40:15.01 81.81 percent complete\n",
      "01:40:34.48 82.08 percent complete\n",
      "01:40:53.75 82.34 percent complete\n",
      "01:41:13.31 82.60 percent complete\n",
      "01:41:32.90 82.86 percent complete\n",
      "01:41:51.63 83.13 percent complete\n",
      "01:42:10.94 83.39 percent complete\n",
      "01:42:30.95 83.65 percent complete\n",
      "01:42:49.94 83.91 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:43:09.27 84.17 percent complete\n",
      "01:43:28.41 84.44 percent complete\n",
      "01:43:46.77 84.70 percent complete\n",
      "01:44:05.03 84.96 percent complete\n",
      "01:44:23.98 85.22 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:44:44.11 85.49 percent complete\n",
      "01:45:03.60 85.75 percent complete\n",
      "01:45:22.51 86.01 percent complete\n",
      "01:45:42.00 86.27 percent complete\n",
      "01:46:00.42 86.53 percent complete\n",
      "01:46:18.52 86.80 percent complete\n",
      "01:46:37.98 87.06 percent complete\n",
      "01:46:56.79 87.32 percent complete\n",
      "01:47:15.61 87.58 percent complete\n",
      "01:47:35.69 87.85 percent complete\n",
      "01:47:54.51 88.11 percent complete\n",
      "01:48:13.05 88.37 percent complete\n",
      "01:48:31.78 88.63 percent complete\n",
      "01:48:50.60 88.89 percent complete\n",
      "01:49:10.00 89.16 percent complete\n",
      "01:49:28.75 89.42 percent complete\n",
      "01:49:48.13 89.68 percent complete\n",
      "01:50:07.42 89.94 percent complete\n",
      "01:50:26.81 90.21 percent complete\n",
      "01:50:46.23 90.47 percent complete\n",
      "01:51:04.90 90.73 percent complete\n",
      "01:51:23.89 90.99 percent complete\n",
      "01:51:43.02 91.25 percent complete\n",
      "01:52:01.42 91.52 percent complete\n",
      "01:52:20.45 91.78 percent complete\n",
      "01:52:40.07 92.04 percent complete\n",
      "01:52:59.13 92.30 percent complete\n",
      "01:53:18.26 92.57 percent complete\n",
      "01:53:37.68 92.83 percent complete\n",
      "01:53:56.38 93.09 percent complete\n",
      "01:54:15.27 93.35 percent complete\n",
      "01:54:34.29 93.61 percent complete\n",
      "01:54:52.86 93.88 percent complete\n",
      "01:55:12.53 94.14 percent complete\n",
      "01:55:32.24 94.40 percent complete\n",
      "01:55:51.31 94.66 percent complete\n",
      "01:56:09.90 94.93 percent complete\n",
      "01:56:28.72 95.19 percent complete\n",
      "01:56:47.90 95.45 percent complete\n",
      "01:57:07.58 95.71 percent complete\n",
      "01:57:26.72 95.97 percent complete\n",
      "01:57:46.36 96.24 percent complete\n",
      "01:58:05.39 96.50 percent complete\n",
      "01:58:24.00 96.76 percent complete\n",
      "01:58:43.78 97.02 percent complete\n",
      "01:59:02.51 97.29 percent complete\n",
      "01:59:21.18 97.55 percent complete\n",
      "01:59:40.21 97.81 percent complete\n",
      "01:59:58.95 98.07 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:00:18.98 98.33 percent complete\n",
      "02:00:37.74 98.60 percent complete\n",
      "02:00:56.97 98.86 percent complete\n",
      "02:01:15.57 99.12 percent complete\n",
      "02:01:34.42 99.38 percent complete\n",
      "02:01:52.82 99.65 percent complete\n",
      "02:02:11.54 99.91 percent complete\n"
     ]
    }
   ],
   "source": [
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_pp.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan\n",
    "\n",
    "# NOTE - This might run slow depending on the size of your training set. We are\n",
    "# printing some information to help you track how long it would take. \n",
    "scores = []\n",
    "start_time = time.time()\n",
    "for idx, row in df_pp.iterrows():\n",
    "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
    "  if idx % 1000 == 0:\n",
    "    hours, rem = divmod(time.time() - start_time, 3600)\n",
    "    minutes, seconds = divmod(rem, 60)\n",
    "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
    "\n",
    "# Filter out \"almost overlapping samples\"\n",
    "df_pp['scores'] = scores\n",
    "df_pp = df_pp[df_pp['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 836
    },
    "colab_type": "code",
    "id": "hxxBOCA-xXhy",
    "outputId": "88a29045-1127-4f7a-8550-cc33ff8b49c7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "christian elders train younger men in the congregation\n",
      "\"in japan , for example , taro had from childhood centered his life on being loyal and obedient to his parents .\"\n",
      "consider what the bible teaches .\n",
      "\"[ map on page 8 , 9 ]\"\n",
      "true friendship is a wonderful gift from jehovah god .\n",
      "7 : 12 .\n",
      "\"the apostle john wrote : “ we know we originate with god , but the whole world is lying in the power of the wicked one . ”\"\n",
      "\"three years later , i was baptized .\"\n",
      "it makes me feel sick .\n",
      "this has occurred in “ the time of the end ” ​ — the time in which we are now living .\n",
      "\n",
      "==> train.ti <==\n",
      "ክርስትያን ሽማግለታት ነቶም ኣብ ጉባኤ ዘለዉ መንእሰያት የሰልጥኑ\n",
      "ንኣብነት ፡ ኣብ ጃፓን ዚነብር ታሮ ኻብ ንእስነቱ ኣትሒዙ ንወለዱ እሙንን እዙዝን እዩ ነይሩ ።\n",
      "መጽሓፍ ቅዱስ እንታይ ከም ዚምህር እስከ ንርአ ።\n",
      "\"[ ኣብ ገጽ 8 , 9 ዘሎ ካርታ ]\"\n",
      "ሓቀኛ ዕርክነት ዘደንቕ ዝዀነ ውህበት የሆዋ ኣምላኽ ኢዩ ።\n",
      "7 : 12 ።\n",
      "ሃዋርያ ዮሃንስ “ ካብ ኣምላኽ ምዃንና : ብዘላ ዓለምውን በቲ እኩይ ተትሒዛ ምህላዋ ንፈልጥ ኢና ” ኢሉ ጸሓፈ ።\n",
      "ድሕሪ ሰለስተ ዓመት ድማ ተጠመቕኩ ።\n",
      "ቅጭ እዩ ዜምጽኣለይ ።\n",
      "እዚ ድማ ኣብዚ እንነብረሉ ዘሎና “ ዘመን መወዳእታ ” ኢዩ ተፈጺሙ ።\n",
      "==> dev.en <==\n",
      "\"long before jesus came to earth , jehovah’s mildness was manifested when cain and abel , adam’s sons , presented sacrifices to god .\"\n",
      "\"show me a sign of your goodness,so that those who hate me may see it and be put to shame. for you, o jehovah, are my helper and comforter.\"\n",
      "\"as jesus made plain in the sermon on the mount , however , even imperfect people can be perfect in a relative sense .\"\n",
      "what made the apostle so sure that jesus was alive ?\n",
      "\"no more food shortages , no one undernourished , nobody starving !\"\n",
      "it is time to seek the help of christian elders .\n",
      "she reports : “ sometimes my classmates ask me questions and even ask for a copy of the book . ”\n",
      "\"a number of them lost family members , breadwinners , friends , employment , or whatever sense of security they thought they had .\"\n",
      "how should a christian treat his marriage partner in private and in the presence of others ?\n",
      "\"aided by god’s spirit , the apostles and older men concluded that obligatory circumcision ended with the law . still , certain divine requirements remained for christians .\"\n",
      "\n",
      "==> dev.ti <==\n",
      "ካብቲ የሱስ ኣብ ምድሪ ዝመጸሉ ነዊሕ ዓመታት ኣቐዲሙ : እቶም ቃየልን ኣቤልን ዝበሃሉ ደቂ ኣዳም መስዋእቲ ከቕርቡ ኸለዉ ለውሃት የሆዋ ተራእዩ እዩ ።\n",
      "ኦ የሆዋ፡ ረዳእየይን መጸናንዕየይን ኢኻ እሞ፡እቶም ዚጸልኡኒ ርእዮም ኪሓፍሩስ፡ምልክት ሕያውነትካ ኣርእየኒ።\n",
      "ኣብቲ ስብከቱ ፡ መለክዒታት እታ ፍጽምቲ ወይ ምልእቲ ፍቕሪ እንታይ ምዃኑ ገለጸ ።\n",
      "ብኸመይ እዩ ርግጸኛ ኪኸውን ክኢሉ ፧\n",
      "ብሰንኪ ሕጽረት ምግቢ ጾሙ ዚሓድር ኣይኪህሉን እዩ ።\n",
      "ካብ ክርስትያን ሽማግለታት ሓገዝ ኪደሊ ኣለዎ ።\n",
      "“ ሓድሓደ ግዜ ደቂ ኽፍለይ ሕቶታት ይሓቱኒ : አረ ቕዳሕ እታ መጽሓፍ ከምጽኣሎም እውን ይሓቱኒ እዮም ” በለት ።\n",
      "ኣብ ሓንቲ ሃገር ግብረ⁠ - ​ ሽበራውያን መጥቃዕቲ ምስ ፈጸሙ : ብዙሓት ሰባት ኣዝዮም ሰምበዱ ።\n",
      "ሓደ ክርስትያን ብብሕቲ ዀነ ኣብ ቅድሚ ሰባት ንብዓልቲ ቤቱ ኸመይ ገይሩ ኺሕዛ ኣለዎ : ሓንቲ ክርስትያን ሰበይቲኸ ፧\n",
      "እቶም ሃዋርያትን ሽማግለታትን እቲ ሕጊ ኻብ ዜብቅዓሉ እዋን ኣትሒዙ ግዝረት ግዴታ ኸም ዘይኰነ ብሓገዝ መንፈስ ቅዱስ ደምደሙ ።\n"
     ]
    }
   ],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "  df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "  df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "iBRMm4kMxZ8L",
    "outputId": "bfcfa301-447a-41b2-8cda-d0c5f49f93a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'joeynmt'...\n",
      "remote: Enumerating objects: 20, done.\u001b[K\n",
      "remote: Counting objects:   5% (1/20)\u001b[K\r",
      "remote: Counting objects:  10% (2/20)\u001b[K\r",
      "remote: Counting objects:  15% (3/20)\u001b[K\r",
      "remote: Counting objects:  20% (4/20)\u001b[K\r",
      "remote: Counting objects:  25% (5/20)\u001b[K\r",
      "remote: Counting objects:  30% (6/20)\u001b[K\r",
      "remote: Counting objects:  35% (7/20)\u001b[K\r",
      "remote: Counting objects:  40% (8/20)\u001b[K\r",
      "remote: Counting objects:  45% (9/20)\u001b[K\r",
      "remote: Counting objects:  50% (10/20)\u001b[K\r",
      "remote: Counting objects:  55% (11/20)\u001b[K\r",
      "remote: Counting objects:  60% (12/20)\u001b[K\r",
      "remote: Counting objects:  65% (13/20)\u001b[K\r",
      "remote: Counting objects:  70% (14/20)\u001b[K\r",
      "remote: Counting objects:  75% (15/20)\u001b[K\r",
      "remote: Counting objects:  80% (16/20)\u001b[K\r",
      "remote: Counting objects:  85% (17/20)\u001b[K\r",
      "remote: Counting objects:  90% (18/20)\u001b[K\r",
      "remote: Counting objects:  95% (19/20)\u001b[K\r",
      "remote: Counting objects: 100% (20/20)\u001b[K\r",
      "remote: Counting objects: 100% (20/20), done.\u001b[K\n",
      "remote: Compressing objects: 100% (17/17), done.\u001b[K\n",
      "remote: Total 2204 (delta 8), reused 5 (delta 3), pack-reused 2184\u001b[K\n",
      "Receiving objects: 100% (2204/2204), 2.60 MiB | 16.87 MiB/s, done.\n",
      "Resolving deltas: 100% (1529/1529), done.\n",
      "Processing /content/joeynmt\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
      "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (6.2.2)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.5)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (42.0.2)\n",
      "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
      "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
      "Collecting sacrebleu>=1.3.6\n",
      "  Downloading https://files.pythonhosted.org/packages/45/31/1a135b964c169984b27fb2f7a50280fa7f8e6d9d404d8a9e596180487fd1/sacrebleu-1.4.3-py3-none-any.whl\n",
      "Collecting subword-nmt\n",
      "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.2)\n",
      "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
      "Collecting pyyaml>=5.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)\n",
      "\u001b[K     |████████████████████████████████| 276kB 9.9MB/s \n",
      "\u001b[?25hCollecting pylint\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
      "\u001b[K     |████████████████████████████████| 307kB 62.2MB/s \n",
      "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
      "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
      "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
      "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
      "Collecting portalocker\n",
      "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
      "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.6)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.4.1)\n",
      "Collecting astroid<2.4,>=2.3.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
      "\u001b[K     |████████████████████████████████| 215kB 73.1MB/s \n",
      "\u001b[?25hCollecting isort<5,>=4.2.5\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 8.8MB/s \n",
      "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
      "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.11.28)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
      "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
      "\u001b[K     |████████████████████████████████| 747kB 61.5MB/s \n",
      "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
      "\u001b[K     |████████████████████████████████| 61kB 10.4MB/s \n",
      "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
      "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=67549da08a638586439b5c2f3d159e654db46437decdb3c467bff46c9c19fe8d\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-gr006ni6/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
      "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for pyyaml: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=44229 sha256=68139519cccf1d9416eb59dcb93f1adfb374a3715c462053dfea8edffc5e2476\n",
      "  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191\n",
      "Successfully built joeynmt pyyaml\n",
      "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, typed-ast, lazy-object-proxy, astroid, isort, mccabe, pylint, joeynmt\n",
      "  Found existing installation: PyYAML 3.13\n",
      "    Uninstalling PyYAML-3.13:\n",
      "      Successfully uninstalled PyYAML-3.13\n",
      "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.3 sacrebleu-1.4.3 subword-nmt-0.3.7 typed-ast-1.4.1\n"
     ]
    }
   ],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip3 install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 436
    },
    "colab_type": "code",
    "id": "H-TyjtmXB1mL",
    "outputId": "f83ab25f-c75a-42fb-b35e-68d3dbdf8670"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.ti     test.ti\t   train.en\n",
      "dev.bpe.en\tdev.ti\t     test.en\t     train.bpe.en  train.ti\n",
      "dev.bpe.ti\ttest.bpe.en  test.en-any.en  train.bpe.ti\n",
      "bpe.codes.4000\tdev.en\t     test.bpe.ti     test.ti\t   train.en\n",
      "dev.bpe.en\tdev.ti\t     test.en\t     train.bpe.en  train.ti\n",
      "dev.bpe.ti\ttest.bpe.en  test.en-any.en  train.bpe.ti\n",
      "BPE Xhosa Sentences\n",
      "ዓብዪ ዋ@@ ል@@ ታ እምነት ( ሕ@@ ጡ@@ ብ ጽሑፍ 12 - 14 ርአ )\n",
      "ናይ ድሕ@@ ነት ቍ@@ ራ@@ ዕ ርእሲ ( ሕ@@ ጡ@@ ብ ጽሑፍ 15 - 18 ርአ )\n",
      "ሰባት ን@@ መጽሓፍ ቅዱስ ከም እት@@ ፈት@@ ዎን ንምሕ@@ ጋ@@ ዞም ዚ@@ ከ@@ ኣለ@@ ካ ት@@ ገብር ኸም ዘለ@@ ኻን ምስ ረኣ@@ ዩ ፡ ጽቡቕ ምላሽ ይ@@ ህ@@ ቡ እዮም ” ይብል ።\n",
      "ናይ መንፈስ ሰይ@@ ፊ ( ሕ@@ ጡ@@ ብ ጽሑፍ 19 - 20 ርአ )\n",
      "ብሓ@@ ገ@@ ዝ የሆዋ ፡ ክን@@ ቃ@@ ወ@@ ሞ ንኽእል ኢና !\n",
      "Combined BPE Vocab\n",
      "ù\n",
      "@\n",
      "➌\n",
      "ā\n",
      "ź\n",
      "#\n",
      "ʺ@@\n",
      "ዃ\n",
      "ṭ@@\n",
      "እቆ@@\n"
     ]
    }
   ],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Xhosa Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IlMitUHR8Qy-"
   },
   "outputs": [],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIs1lY2hxMsl"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (source_language, target_language)\n",
    "gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 70                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_transformer\"\n",
    "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 8             # Increased to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 512   # Increased to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 512         # Increased to 512 for larger data.\n",
    "        ff_size: 2048            # Increased to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 8              # Increased to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 512    # Increased to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 512         # Increased to 512 for larger data.\n",
    "        ff_size: 2048            # Increased to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
    "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pIifxE3Qzuvs"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "6ZBPFwT94WpI",
    "outputId": "2bb0fc1e-cf9a-4138-c82a-1f17118886ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-01-21 13:43:24,056 Hello! This is Joey-NMT.\n",
      "2020-01-21 13:43:25,233 Total params: 46600192\n",
      "2020-01-21 13:43:25,234 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
      "2020-01-21 13:43:34,451 cfg.name                           : enti_transformer\n",
      "2020-01-21 13:43:34,451 cfg.data.src                       : en\n",
      "2020-01-21 13:43:34,451 cfg.data.trg                       : ti\n",
      "2020-01-21 13:43:34,451 cfg.data.train                     : data/enti/train.bpe\n",
      "2020-01-21 13:43:34,451 cfg.data.dev                       : data/enti/dev.bpe\n",
      "2020-01-21 13:43:34,451 cfg.data.test                      : data/enti/test.bpe\n",
      "2020-01-21 13:43:34,451 cfg.data.level                     : bpe\n",
      "2020-01-21 13:43:34,451 cfg.data.lowercase                 : False\n",
      "2020-01-21 13:43:34,451 cfg.data.max_sent_length           : 100\n",
      "2020-01-21 13:43:34,451 cfg.data.src_vocab                 : data/enti/vocab.txt\n",
      "2020-01-21 13:43:34,451 cfg.data.trg_vocab                 : data/enti/vocab.txt\n",
      "2020-01-21 13:43:34,451 cfg.testing.beam_size              : 5\n",
      "2020-01-21 13:43:34,451 cfg.testing.alpha                  : 1.0\n",
      "2020-01-21 13:43:34,451 cfg.training.random_seed           : 42\n",
      "2020-01-21 13:43:34,451 cfg.training.optimizer             : adam\n",
      "2020-01-21 13:43:34,451 cfg.training.normalization         : tokens\n",
      "2020-01-21 13:43:34,451 cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2020-01-21 13:43:34,452 cfg.training.scheduling            : plateau\n",
      "2020-01-21 13:43:34,452 cfg.training.patience              : 5\n",
      "2020-01-21 13:43:34,452 cfg.training.learning_rate_factor  : 0.5\n",
      "2020-01-21 13:43:34,452 cfg.training.learning_rate_warmup  : 1000\n",
      "2020-01-21 13:43:34,452 cfg.training.decrease_factor       : 0.7\n",
      "2020-01-21 13:43:34,452 cfg.training.loss                  : crossentropy\n",
      "2020-01-21 13:43:34,452 cfg.training.learning_rate         : 0.0003\n",
      "2020-01-21 13:43:34,452 cfg.training.learning_rate_min     : 1e-08\n",
      "2020-01-21 13:43:34,452 cfg.training.weight_decay          : 0.0\n",
      "2020-01-21 13:43:34,452 cfg.training.label_smoothing       : 0.1\n",
      "2020-01-21 13:43:34,452 cfg.training.batch_size            : 4096\n",
      "2020-01-21 13:43:34,452 cfg.training.batch_type            : token\n",
      "2020-01-21 13:43:34,452 cfg.training.eval_batch_size       : 3600\n",
      "2020-01-21 13:43:34,452 cfg.training.eval_batch_type       : token\n",
      "2020-01-21 13:43:34,452 cfg.training.batch_multiplier      : 1\n",
      "2020-01-21 13:43:34,452 cfg.training.early_stopping_metric : ppl\n",
      "2020-01-21 13:43:34,452 cfg.training.epochs                : 30\n",
      "2020-01-21 13:43:34,452 cfg.training.validation_freq       : 1000\n",
      "2020-01-21 13:43:34,452 cfg.training.logging_freq          : 100\n",
      "2020-01-21 13:43:34,452 cfg.training.eval_metric           : bleu\n",
      "2020-01-21 13:43:34,452 cfg.training.model_dir             : models/enti_transformer\n",
      "2020-01-21 13:43:34,452 cfg.training.overwrite             : False\n",
      "2020-01-21 13:43:34,452 cfg.training.shuffle               : True\n",
      "2020-01-21 13:43:34,452 cfg.training.use_cuda              : True\n",
      "2020-01-21 13:43:34,453 cfg.training.max_output_length     : 100\n",
      "2020-01-21 13:43:34,453 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2020-01-21 13:43:34,453 cfg.training.keep_last_ckpts       : 3\n",
      "2020-01-21 13:43:34,453 cfg.model.initializer              : xavier\n",
      "2020-01-21 13:43:34,453 cfg.model.bias_initializer         : zeros\n",
      "2020-01-21 13:43:34,453 cfg.model.init_gain                : 1.0\n",
      "2020-01-21 13:43:34,453 cfg.model.embed_initializer        : xavier\n",
      "2020-01-21 13:43:34,453 cfg.model.embed_init_gain          : 1.0\n",
      "2020-01-21 13:43:34,453 cfg.model.tied_embeddings          : True\n",
      "2020-01-21 13:43:34,453 cfg.model.tied_softmax             : True\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.type             : transformer\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.num_layers       : 6\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.num_heads        : 8\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.embeddings.embedding_dim : 512\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.embeddings.scale : True\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.hidden_size      : 512\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.ff_size          : 2048\n",
      "2020-01-21 13:43:34,453 cfg.model.encoder.dropout          : 0.3\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.type             : transformer\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.num_layers       : 6\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.num_heads        : 8\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.embeddings.embedding_dim : 512\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.embeddings.scale : True\n",
      "2020-01-21 13:43:34,453 cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2020-01-21 13:43:34,454 cfg.model.decoder.hidden_size      : 512\n",
      "2020-01-21 13:43:34,454 cfg.model.decoder.ff_size          : 2048\n",
      "2020-01-21 13:43:34,454 cfg.model.decoder.dropout          : 0.3\n",
      "2020-01-21 13:43:34,454 Data set sizes: \n",
      "\ttrain 378230,\n",
      "\tvalid 1000,\n",
      "\ttest 2618\n",
      "2020-01-21 13:43:34,454 First training example:\n",
      "\t[SRC] christian elders tra@@ in youn@@ ger men in the congregation\n",
      "\t[TRG] ክርስትያን ሽማግለታት ነቶም ኣብ ጉባኤ ዘለዉ መንእሰያት የ@@ ሰል@@ ጥ@@ ኑ\n",
      "2020-01-21 13:43:34,454 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) the (5) , (6) ። (7) . (8) to (9) :\n",
      "2020-01-21 13:43:34,454 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) the (5) , (6) ። (7) . (8) to (9) :\n",
      "2020-01-21 13:43:34,454 Number of Src words (types): 4804\n",
      "2020-01-21 13:43:34,455 Number of Trg words (types): 4804\n",
      "2020-01-21 13:43:34,455 Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=8),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=8),\n",
      "\tsrc_embed=Embeddings(embedding_dim=512, vocab_size=4804),\n",
      "\ttrg_embed=Embeddings(embedding_dim=512, vocab_size=4804))\n",
      "2020-01-21 13:43:34,459 EPOCH 1\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n",
      "    \"__main__\", mod_spec)\n",
      "  File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/content/joeynmt/joeynmt/__main__.py\", line 41, in <module>\n",
      "    main()\n",
      "  File \"/content/joeynmt/joeynmt/__main__.py\", line 29, in main\n",
      "    train(cfg_file=args.config_path)\n",
      "  File \"/content/joeynmt/joeynmt/training.py\", line 596, in train\n",
      "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
      "  File \"/content/joeynmt/joeynmt/training.py\", line 296, in train_and_validate\n",
      "    batch_loss = self._train_batch(batch, update=update)\n",
      "  File \"/content/joeynmt/joeynmt/training.py\", line 459, in _train_batch\n",
      "    self.optimizer.step()\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py\", line 95, in step\n",
      "    exp_avg.mul_(beta1).add_(1 - beta1, grad)\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n94wlrCjVc17"
   },
   "outputs": [],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "66WhRE9lIhoD"
   },
   "outputs": [],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "starter_notebook.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3.7",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}