en-nso/autshumato-baseline/ennso.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "ennso.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "166cc992-a421-4214-e5bf-7fdb0b97f537",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"nso\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "7ab8e019-4c5f-4277-f3e4-07b413c74d79",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-nso-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "4daa0b2b-df93-4c0c-a4d4-a2dcd3b08be1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 12.2MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 5.0MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 6.9MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 8.8MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 4.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 4.9MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 5.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 3.9MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "colab": {}
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "outputId": "828549e6-2c48-45d9-f107-1d661eb2915d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-11-26 15:28:50--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2019-11-26 15:28:50 (11.9 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2019-11-26 15:28:54--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nso.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 205401 (201K) [text/plain]\n",
            "Saving to: ‘test.en-nso.en’\n",
            "\n",
            "test.en-nso.en      100%[===================>] 200.59K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2019-11-26 15:28:54 (11.4 MB/s) - ‘test.en-nso.en’ saved [205401/205401]\n",
            "\n",
            "--2019-11-26 15:29:01--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nso.nso\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 244175 (238K) [text/plain]\n",
            "Saving to: ‘test.en-nso.nso’\n",
            "\n",
            "test.en-nso.nso     100%[===================>] 238.45K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2019-11-26 15:29:02 (10.5 MB/s) - ‘test.en-nso.nso’ saved [244175/244175]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "outputId": "f13c3aca-7a49-4149-d252-2b57c5c01fb2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "abf8c62d-efb5-4aea-84db-83dbcb594fa3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'ennso_parallel.train.' + source_language\n",
        "target_file = 'ennso_parallel.train.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 0/27776 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Managing the Groote Schuur and Red Cross Child...</td>\n",
              "      <td>Go laola  Groote Schuur le karolo ya akademi y...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Furnace burners for liquid fuel, for pulverise...</td>\n",
              "      <td>Dipenara diruthufatši, bakeng sa dibešwa tša d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Moreover Stats SA re-demarcated cross-boundary...</td>\n",
              "      <td>Ke ka baka leo  Stats SA e kgaogantšeleswa meb...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0  Managing the Groote Schuur and Red Cross Child...  Go laola  Groote Schuur le karolo ya akademi y...\n",
              "1  Furnace burners for liquid fuel, for pulverise...  Dipenara diruthufatši, bakeng sa dibešwa tša d...\n",
              "2  Moreover Stats SA re-demarcated cross-boundary...  Ke ka baka leo  Stats SA e kgaogantšeleswa meb..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "ee9ddbf6-d8a5-4650-e926-660ca841cb48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "outputId": "aaa80300-4b40-4e1c-d6b8-ca564ebd78a4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 742
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.17.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 3.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (41.6.0)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144667 sha256=bd728b6572a155e537f5b5bdd8f64537864999ba3d6830247c34be63ab68f52e\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n",
            "00:00:00.03 0.00 percent complete\n",
            "00:00:23.57 3.81 percent complete\n",
            "00:00:47.13 7.61 percent complete\n",
            "00:01:10.65 11.42 percent complete\n",
            "00:01:34.98 15.22 percent complete\n",
            "00:01:59.15 19.03 percent complete\n",
            "00:02:22.49 22.83 percent complete\n",
            "00:02:45.51 26.64 percent complete\n",
            "00:03:09.27 30.44 percent complete\n",
            "00:03:33.89 34.25 percent complete\n",
            "00:03:57.13 38.05 percent complete\n",
            "00:04:20.65 41.86 percent complete\n",
            "00:04:44.13 45.67 percent complete\n",
            "00:05:07.52 49.47 percent complete\n",
            "00:05:31.28 53.28 percent complete\n",
            "00:05:55.02 57.08 percent complete\n",
            "00:06:17.48 60.89 percent complete\n",
            "00:06:40.76 64.69 percent complete\n",
            "00:07:04.37 68.50 percent complete\n",
            "00:07:28.14 72.30 percent complete\n",
            "00:07:51.38 76.11 percent complete\n",
            "00:08:14.92 79.91 percent complete\n",
            "00:08:38.80 83.72 percent complete\n",
            "00:09:01.61 87.53 percent complete\n",
            "00:09:24.94 91.33 percent complete\n",
            "00:09:49.48 95.14 percent complete\n",
            "00:10:12.77 98.94 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "outputId": "1decc3e0-fd2b-41a1-b941-6038da60c551",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 810
        }
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "Woodchips were deposited to keep moist on the ground and prevent wind from blowing the newly deposited sand away.\n",
            "It is ideally suited to small businesses .\n",
            "Applications will be considered by the WCCC during July and successful applicants be notified and first payments made by the end of August.\n",
            "seed or propagating material\n",
            "A line or column must be added to the milk recording book, or the cow byre sheet, for the regular recording of cow condition.\n",
            "Ancient Egyptians invented mathematics and divided it into arithmetic, algebra and geometry.\n",
            "However, Nicholas Burns, the US undersecretary of state, characterised the deals as a continuation of existing policy.\n",
            "Victims of domestic violence who are in a crisis situation can stay at a shelter for a period of between two weeks and six months depending on their circumstances.\n",
            "If you are concerned that you may have a gambling problem, try completing the Responsible Gaming online checklist.\n",
            "a list of publications\n",
            "\n",
            "==> train.nso <==\n",
            "Ditshetlana tša dikota di be di tšhelwa fase go boloka lebala gore le dule le kolobile le go thibela go foka ga phefo ye e ka rwalago santa ye e sa tšwago go tšhelwa.\n",
            "E swanetše dikgwebo tše nnyane .\n",
            "Dikgopelo di tla dumelelwa ke WCCC ka Julae gomme bakgopedi bao ba atlegilego ba tla tsebišwa gomme ditefelo tša mathomo tša dirwa mafefelong a Agostose.\n",
            "peu goba materiale wo o tswadišago\n",
            "Mothaladi goba kholomo e swanetše go tsenywa ka pukung ya go bega maswi, goba llaga ya mmutedi ya ka lešakeng la kgomo ya tshadi, go tlo gatiša seemo sa kgomo ya tshadi kgafetšakgafetša.\n",
            "Maegepeta a kgalekgale a dirile mmetse gomme a o arola ka arithimethiki, altšipra le tšeomethri.\n",
            "Le ge go le bjale, Nicholas Burns, mongwaledi wa ka fasana wa mmušo wa US, o bone ditirišano tše bjalo ka tšwetšopele ya melawana ye e lego gona.\n",
            "Batšwasehlabelo ba bohlokotsebe bja ka malapeng bao ba lego seemong sa matšhošetši ba ka dula lefelong la tšhireletšo lebaka la magareng ga dibeke tše pedi le dikgwedi tše tshela go e ya ka seemo sa bona.\n",
            "Ge e le gore o a tshwenyega gore o ka be o na le bothata bja go betša, leka go tlatša lenaneo la dilo tše di nyakegago la inthaneteng la mabapi le Go Betša ka Maikarabelo.\n",
            "lenaneo la dikgatišo\n",
            "==> dev.en <==\n",
            "A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "Application for resumption of South African citizenship, BI-175\n",
            "Our policy is to provide information as and when it becomes available.\n",
            "We have learnt a lot about what it takes to run a good mother-to-child HIV prevention programme.\n",
            "Various global events have been smoothly staged, including the Cricket World Cup opening and the President's Golf Cup.\n",
            "General repairs and renovations: Glendale Senior Secondary School, Mitchell's Plain.\n",
            "These issue-driven, comedy-dramas have a huge following and are performed in many community theatres and improvise theatre venues on the outskirts of Durban.\n",
            "In terms of the Regulation of Gatherings Act, we cannot unreasonably limit the right of the Minstrels to hold public gatherings.\n",
            "This handy booklet on SoE reporting was developed by Grid Arendal.\n",
            "\n",
            "==> dev.nso <==\n",
            "Bosenyi kgahlanong le botho, bo ka se efogwe  gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "Application for resumption of South African citizenship, BI-175\n",
            "Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "Re ithutile tše ntši ka seo se nyakegago go sepediša lenaneo la thibelo ya HIV go tloga go motswadi go ya go ngwana.\n",
            "Ke rata go fetiša mantšu aka a ditebogo go ba karolo ya tiragalo ye kgolo ye le go kgona go hloma tirelo ya segwera ka go Tlhokomelo ya Maphelo ya Motheo mo go la Welkom lehono.\n",
            "Ditokišokakaretšo le dimpshafatšo: Sekolo se se Phagamego sa Godingwana sa Glendale, ka Mitchell's Plain.\n",
            "Dipapadi tše tša metlae tše di theilwego go ditaba di latelwa kudu gomme di diragatšwa ka ditheethareng tše ntši tša setšhaba le go kaonafatša mafelo a theetha ka mathomong a Durban.\n",
            "Go ya ka Molao wa Taolo ya Dikgobokano, re ka se direle tokelo ya Minstrels ya go swara dikgobokano tša setšhaba mellwane.\n",
            "Pukwana ye ya go šikarwa ya pego ya SoE e tšweleditšwe ke Grid Arendal.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "outputId": "d6a5320f-a4e6-4388-c946-d6f8da9e7830",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 15, done.\u001b[K\n",
            "remote: Counting objects:   6% (1/15)\u001b[K\rremote: Counting objects:  13% (2/15)\u001b[K\rremote: Counting objects:  20% (3/15)\u001b[K\rremote: Counting objects:  26% (4/15)\u001b[K\rremote: Counting objects:  33% (5/15)\u001b[K\rremote: Counting objects:  40% (6/15)\u001b[K\rremote: Counting objects:  46% (7/15)\u001b[K\rremote: Counting objects:  53% (8/15)\u001b[K\rremote: Counting objects:  60% (9/15)\u001b[K\rremote: Counting objects:  66% (10/15)\u001b[K\rremote: Counting objects:  73% (11/15)\u001b[K\rremote: Counting objects:  80% (12/15)\u001b[K\rremote: Counting objects:  86% (13/15)\u001b[K\rremote: Counting objects:  93% (14/15)\u001b[K\rremote: Counting objects: 100% (15/15)\u001b[K\rremote: Counting objects: 100% (15/15), done.\u001b[K\n",
            "remote: Compressing objects: 100% (12/12), done.\u001b[K\n",
            "remote: Total 2199 (delta 4), reused 5 (delta 3), pack-reused 2184\u001b[K\n",
            "Receiving objects: 100% (2199/2199), 2.60 MiB | 4.33 MiB/s, done.\n",
            "Resolving deltas: 100% (1525/1525), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (4.3.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.4)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (41.6.0)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "  Downloading https://files.pythonhosted.org/packages/0e/e5/93d252182f7cbd4b59bb3ec5797e2ce33cfd6f5aadaf327db170cf4b7887/sacrebleu-1.4.2-py3-none-any.whl\n",
            "Collecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)\n",
            "\u001b[K     |████████████████████████████████| 266kB 19.4MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
            "\u001b[K     |████████████████████████████████| 307kB 48.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->joeynmt==0.0.1) (0.46)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.5)\n",
            "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.3.2)\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 7.2MB/s \n",
            "\u001b[?25hCollecting astroid<2.4,>=2.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 40.1MB/s \n",
            "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.9.11)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 7.0MB/s \n",
            "\u001b[?25hCollecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/31/d3/9d1802c161626d0278bafb1ffb32f76b9d01e123881bbf9d91e8ccf28e18/typed_ast-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (736kB)\n",
            "\u001b[K     |████████████████████████████████| 737kB 46.7MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=4204f0accd4fe7b576bea22cfe3ab1383d73cf9d4d4b03d93247e2697a5a8976\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-n9999f2h/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=a58f1ec793041964ea4df3291b4a33b18bc47483dd197d6f6198fc8f3e7b0d4e\n",
            "  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030\n",
            "Successfully built joeynmt pyyaml\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, isort, lazy-object-proxy, typed-ast, astroid, mccabe, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.1.2 sacrebleu-1.4.2 subword-nmt-0.3.7 typed-ast-1.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "outputId": "d90708c9-e3b2-4cc8-979d-75607f24739d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        }
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.4000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Northern Sotho Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.en\t     test.bpe.nso    test.nso\t    train.en\n",
            "dev.bpe.en\tdev.nso      test.en\t     train.bpe.en   train.nso\n",
            "dev.bpe.nso\ttest.bpe.en  test.en-any.en  train.bpe.nso\n",
            "bpe.codes.4000\tdev.en\t test.bpe.en   test.en-any.en  train.bpe.nso\n",
            "dev.bpe.en\tdev.nso  test.bpe.nso  test.nso        train.en\n",
            "dev.bpe.nso\tmodels\t test.en       train.bpe.en    train.nso\n",
            "BPE Northern Sotho Sentences\n",
            "Maf@@ elelong ke ile ka tum@@ a ka go se bo@@ te@@ ge .\n",
            "Ka morago ga gore ke ithut@@ e there@@ šo , ke ile ka g@@ ana go tšwela pele ke dira seo , ga@@ e@@ š@@ it@@ a le ge mošomo woo o be o lef@@ a gabotse .\n",
            "Ke mohlala o mo@@ botse go bar@@ wa ba ka ba babedi , e bile ke ne@@ ilwe ditokelo ka phu@@ the@@ gong .\n",
            "G@@ ona bjale ke tum@@ ile ka go bo@@ te@@ ga gare ga ba@@ hlahlo@@ bi ba ma@@ kgetho le bao ke dirago kgwebo le bona . ”\n",
            "R@@ u@@ the o ile a hu@@ du@@ g@@ ela I@@ si@@ ra@@ ele , moo a bego a ka rap@@ ela Mo@@ dimo wa there@@ šo .\n",
            "Combined BPE Vocab\n",
            "′\n",
            "espec@@\n",
            "=\n",
            "é@@\n",
            "—\n",
            "ween\n",
            "ï@@\n",
            "<\n",
            "\\@@\n",
            "ingwa@@\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlMitUHR8Qy-",
        "outputId": "fb1d3877-3a6f-4de3-f0bc-b3150fd38a0f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\""
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.en\t test.bpe.en   test.en-any.en  train.bpe.nso\n",
            "dev.bpe.en\tdev.nso  test.bpe.nso  test.nso        train.en\n",
            "dev.bpe.nso\tmodels\t test.en       train.bpe.en    train.nso\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"data/{name}/train.bpe\"\n",
        "    dev:   \"data/{name}/dev.bpe\"\n",
        "    test:  \"data/{name}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"data/{name}/vocab.txt\"\n",
        "    trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 100                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 2000          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "outputId": "c8b9ac1c-d6cb-4757-b81a-40aacb496fb8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-11-26 15:42:24,954 Hello! This is Joey-NMT.\n",
            "2019-11-26 15:42:26,730 Total params: 12132864\n",
            "2019-11-26 15:42:26,732 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2019-11-26 15:42:32,060 cfg.name                           : ennso_transformer\n",
            "2019-11-26 15:42:32,060 cfg.data.src                       : en\n",
            "2019-11-26 15:42:32,060 cfg.data.trg                       : nso\n",
            "2019-11-26 15:42:32,061 cfg.data.train                     : data/ennso/train.bpe\n",
            "2019-11-26 15:42:32,061 cfg.data.dev                       : data/ennso/dev.bpe\n",
            "2019-11-26 15:42:32,061 cfg.data.test                      : data/ennso/test.bpe\n",
            "2019-11-26 15:42:32,061 cfg.data.level                     : bpe\n",
            "2019-11-26 15:42:32,061 cfg.data.lowercase                 : False\n",
            "2019-11-26 15:42:32,061 cfg.data.max_sent_length           : 100\n",
            "2019-11-26 15:42:32,061 cfg.data.src_vocab                 : data/ennso/vocab.txt\n",
            "2019-11-26 15:42:32,061 cfg.data.trg_vocab                 : data/ennso/vocab.txt\n",
            "2019-11-26 15:42:32,061 cfg.testing.beam_size              : 5\n",
            "2019-11-26 15:42:32,061 cfg.testing.alpha                  : 1.0\n",
            "2019-11-26 15:42:32,061 cfg.training.random_seed           : 42\n",
            "2019-11-26 15:42:32,061 cfg.training.optimizer             : adam\n",
            "2019-11-26 15:42:32,061 cfg.training.normalization         : tokens\n",
            "2019-11-26 15:42:32,062 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2019-11-26 15:42:32,062 cfg.training.scheduling            : plateau\n",
            "2019-11-26 15:42:32,062 cfg.training.patience              : 5\n",
            "2019-11-26 15:42:32,062 cfg.training.learning_rate_factor  : 0.5\n",
            "2019-11-26 15:42:32,062 cfg.training.learning_rate_warmup  : 1000\n",
            "2019-11-26 15:42:32,062 cfg.training.decrease_factor       : 0.7\n",
            "2019-11-26 15:42:32,062 cfg.training.loss                  : crossentropy\n",
            "2019-11-26 15:42:32,062 cfg.training.learning_rate         : 0.0003\n",
            "2019-11-26 15:42:32,062 cfg.training.learning_rate_min     : 1e-08\n",
            "2019-11-26 15:42:32,062 cfg.training.weight_decay          : 0.0\n",
            "2019-11-26 15:42:32,062 cfg.training.label_smoothing       : 0.1\n",
            "2019-11-26 15:42:32,062 cfg.training.batch_size            : 4096\n",
            "2019-11-26 15:42:32,062 cfg.training.batch_type            : token\n",
            "2019-11-26 15:42:32,063 cfg.training.eval_batch_size       : 3600\n",
            "2019-11-26 15:42:32,063 cfg.training.eval_batch_type       : token\n",
            "2019-11-26 15:42:32,063 cfg.training.batch_multiplier      : 1\n",
            "2019-11-26 15:42:32,063 cfg.training.early_stopping_metric : ppl\n",
            "2019-11-26 15:42:32,063 cfg.training.epochs                : 100\n",
            "2019-11-26 15:42:32,063 cfg.training.validation_freq       : 2000\n",
            "2019-11-26 15:42:32,063 cfg.training.logging_freq          : 100\n",
            "2019-11-26 15:42:32,063 cfg.training.eval_metric           : bleu\n",
            "2019-11-26 15:42:32,063 cfg.training.model_dir             : models/ennso_transformer\n",
            "2019-11-26 15:42:32,063 cfg.training.overwrite             : False\n",
            "2019-11-26 15:42:32,063 cfg.training.shuffle               : True\n",
            "2019-11-26 15:42:32,063 cfg.training.use_cuda              : True\n",
            "2019-11-26 15:42:32,063 cfg.training.max_output_length     : 100\n",
            "2019-11-26 15:42:32,064 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2019-11-26 15:42:32,064 cfg.training.keep_last_ckpts       : 3\n",
            "2019-11-26 15:42:32,064 cfg.model.initializer              : xavier\n",
            "2019-11-26 15:42:32,064 cfg.model.bias_initializer         : zeros\n",
            "2019-11-26 15:42:32,064 cfg.model.init_gain                : 1.0\n",
            "2019-11-26 15:42:32,064 cfg.model.embed_initializer        : xavier\n",
            "2019-11-26 15:42:32,064 cfg.model.embed_init_gain          : 1.0\n",
            "2019-11-26 15:42:32,064 cfg.model.tied_embeddings          : True\n",
            "2019-11-26 15:42:32,064 cfg.model.tied_softmax             : True\n",
            "2019-11-26 15:42:32,064 cfg.model.encoder.type             : transformer\n",
            "2019-11-26 15:42:32,064 cfg.model.encoder.num_layers       : 6\n",
            "2019-11-26 15:42:32,064 cfg.model.encoder.num_heads        : 4\n",
            "2019-11-26 15:42:32,064 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2019-11-26 15:42:32,064 cfg.model.encoder.embeddings.scale : True\n",
            "2019-11-26 15:42:32,065 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2019-11-26 15:42:32,065 cfg.model.encoder.hidden_size      : 256\n",
            "2019-11-26 15:42:32,065 cfg.model.encoder.ff_size          : 1024\n",
            "2019-11-26 15:42:32,065 cfg.model.encoder.dropout          : 0.3\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.type             : transformer\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.num_layers       : 6\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.num_heads        : 4\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.embeddings.scale : True\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.hidden_size      : 256\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.ff_size          : 1024\n",
            "2019-11-26 15:42:32,065 cfg.model.decoder.dropout          : 0.3\n",
            "2019-11-26 15:42:32,065 Data set sizes: \n",
            "\ttrain 25110,\n",
            "\tvalid 1000,\n",
            "\ttest 2705\n",
            "2019-11-26 15:42:32,065 First training example:\n",
            "\t[SRC] W@@ oo@@ d@@ ch@@ ip@@ s were de@@ po@@ sit@@ ed to ke@@ ep mo@@ ist on the gr@@ ound and prev@@ ent win@@ d from b@@ low@@ ing the ne@@ w@@ ly de@@ po@@ sit@@ ed s@@ and a@@ wa@@ y.\n",
            "\t[TRG] Dit@@ she@@ tl@@ ana tša diko@@ ta di be di tšh@@ elwa fase go boloka leb@@ ala gore le du@@ le le kolo@@ bile le go thibela go fo@@ ka ga phe@@ f@@ o ye e ka r@@ wal@@ ago san@@ ta ye e sa tšwago go tšh@@ elwa.\n",
            "2019-11-26 15:42:32,066 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) go (5) ya (6) the (7) a (8) le (9) ka\n",
            "2019-11-26 15:42:32,066 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) go (5) ya (6) the (7) a (8) le (9) ka\n",
            "2019-11-26 15:42:32,066 Number of Src words (types): 4190\n",
            "2019-11-26 15:42:32,067 Number of Trg words (types): 4190\n",
            "2019-11-26 15:42:32,067 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4190),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4190))\n",
            "2019-11-26 15:42:32,071 EPOCH 1\n",
            "2019-11-26 15:43:03,694 Epoch   1 Step:      100 Batch Loss:     5.566753 Tokens per Sec:     7064, Lr: 0.000300\n",
            "2019-11-26 15:43:35,123 Epoch   1 Step:      200 Batch Loss:     5.344368 Tokens per Sec:     6890, Lr: 0.000300\n",
            "2019-11-26 15:44:06,695 Epoch   1 Step:      300 Batch Loss:     5.020310 Tokens per Sec:     7067, Lr: 0.000300\n",
            "2019-11-26 15:44:30,015 Epoch   1: total training loss 2017.10\n",
            "2019-11-26 15:44:30,016 EPOCH 2\n",
            "2019-11-26 15:44:38,183 Epoch   2 Step:      400 Batch Loss:     4.711702 Tokens per Sec:     6832, Lr: 0.000300\n",
            "2019-11-26 15:45:09,723 Epoch   2 Step:      500 Batch Loss:     4.920614 Tokens per Sec:     7138, Lr: 0.000300\n",
            "2019-11-26 15:45:41,154 Epoch   2 Step:      600 Batch Loss:     4.867137 Tokens per Sec:     6934, Lr: 0.000300\n",
            "2019-11-26 15:46:12,959 Epoch   2 Step:      700 Batch Loss:     4.594979 Tokens per Sec:     7212, Lr: 0.000300\n",
            "2019-11-26 15:46:27,412 Epoch   2: total training loss 1769.68\n",
            "2019-11-26 15:46:27,412 EPOCH 3\n",
            "2019-11-26 15:46:43,983 Epoch   3 Step:      800 Batch Loss:     4.323524 Tokens per Sec:     7022, Lr: 0.000300\n",
            "2019-11-26 15:47:14,970 Epoch   3 Step:      900 Batch Loss:     4.494812 Tokens per Sec:     6990, Lr: 0.000300\n",
            "2019-11-26 15:47:46,373 Epoch   3 Step:     1000 Batch Loss:     4.452847 Tokens per Sec:     7140, Lr: 0.000300\n",
            "2019-11-26 15:48:17,666 Epoch   3 Step:     1100 Batch Loss:     4.147206 Tokens per Sec:     7044, Lr: 0.000300\n",
            "2019-11-26 15:48:25,235 Epoch   3: total training loss 1605.75\n",
            "2019-11-26 15:48:25,235 EPOCH 4\n",
            "2019-11-26 15:48:49,115 Epoch   4 Step:     1200 Batch Loss:     3.981709 Tokens per Sec:     7026, Lr: 0.000300\n",
            "2019-11-26 15:49:20,747 Epoch   4 Step:     1300 Batch Loss:     4.056430 Tokens per Sec:     7095, Lr: 0.000300\n",
            "2019-11-26 15:49:51,857 Epoch   4 Step:     1400 Batch Loss:     3.824705 Tokens per Sec:     6971, Lr: 0.000300\n",
            "2019-11-26 15:50:22,891 Epoch   4: total training loss 1487.38\n",
            "2019-11-26 15:50:22,891 EPOCH 5\n",
            "2019-11-26 15:50:23,276 Epoch   5 Step:     1500 Batch Loss:     3.921676 Tokens per Sec:     7339, Lr: 0.000300\n",
            "2019-11-26 15:50:55,041 Epoch   5 Step:     1600 Batch Loss:     3.524999 Tokens per Sec:     7200, Lr: 0.000300\n",
            "2019-11-26 15:51:26,252 Epoch   5 Step:     1700 Batch Loss:     3.174614 Tokens per Sec:     6964, Lr: 0.000300\n",
            "2019-11-26 15:51:57,818 Epoch   5 Step:     1800 Batch Loss:     3.628934 Tokens per Sec:     7084, Lr: 0.000300\n",
            "2019-11-26 15:52:20,079 Epoch   5: total training loss 1389.27\n",
            "2019-11-26 15:52:20,079 EPOCH 6\n",
            "2019-11-26 15:52:29,349 Epoch   6 Step:     1900 Batch Loss:     3.575956 Tokens per Sec:     6974, Lr: 0.000300\n",
            "2019-11-26 15:53:00,390 Epoch   6 Step:     2000 Batch Loss:     3.547879 Tokens per Sec:     7025, Lr: 0.000300\n",
            "2019-11-26 15:54:41,272 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 15:54:41,272 Saving new checkpoint.\n",
            "2019-11-26 15:54:41,635 Example #0\n",
            "2019-11-26 15:54:41,636 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 15:54:41,636 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 15:54:41,636 \tHypothesis: Go na le go netefatša gore go na le go na le go dira gore go na le go na le go se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se\n",
            "2019-11-26 15:54:41,636 Example #1\n",
            "2019-11-26 15:54:41,636 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 15:54:41,637 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 15:54:41,637 \tHypothesis: Go na le go na le go tšwa go tšwa go tšwa go la go la go tšwa go la go la ka go tšwa go la go la go la ka go la ka go la go la go la ka go tšwa go la go la go la go la ka go la ka go la go la go la ka go la go la ka go la go la ka go la ka go la ka go la go la go la go la go la go la go la go la go la go la go\n",
            "2019-11-26 15:54:41,637 Example #2\n",
            "2019-11-26 15:54:41,637 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 15:54:41,637 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 15:54:41,637 \tHypothesis: Kgopelo ya go ngwadiša go tšwa go Kgoro ya Diphoofolo ya Afrika Borwa ,\n",
            "2019-11-26 15:54:41,637 Example #3\n",
            "2019-11-26 15:54:41,638 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 15:54:41,638 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 15:54:41,638 \tHypothesis: Dikgokagano tša go tšwa go tšwa go go go go go go go dira gore di le gore di ka se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se se\n",
            "2019-11-26 15:54:41,639 Validation result (greedy) at epoch   6, step     2000: bleu:   0.46, loss: 117066.4375, ppl:  34.0924, duration: 101.2476s\n",
            "2019-11-26 15:55:12,763 Epoch   6 Step:     2100 Batch Loss:     3.517848 Tokens per Sec:     7025, Lr: 0.000300\n",
            "2019-11-26 15:55:44,277 Epoch   6 Step:     2200 Batch Loss:     3.587431 Tokens per Sec:     7050, Lr: 0.000300\n",
            "2019-11-26 15:55:59,407 Epoch   6: total training loss 1342.05\n",
            "2019-11-26 15:55:59,408 EPOCH 7\n",
            "2019-11-26 15:56:15,366 Epoch   7 Step:     2300 Batch Loss:     3.666075 Tokens per Sec:     6963, Lr: 0.000300\n",
            "2019-11-26 15:56:46,539 Epoch   7 Step:     2400 Batch Loss:     3.580449 Tokens per Sec:     6914, Lr: 0.000300\n",
            "2019-11-26 15:57:18,077 Epoch   7 Step:     2500 Batch Loss:     3.555959 Tokens per Sec:     7107, Lr: 0.000300\n",
            "2019-11-26 15:57:49,423 Epoch   7 Step:     2600 Batch Loss:     3.336342 Tokens per Sec:     7175, Lr: 0.000300\n",
            "2019-11-26 15:57:57,395 Epoch   7: total training loss 1288.50\n",
            "2019-11-26 15:57:57,395 EPOCH 8\n",
            "2019-11-26 15:58:20,340 Epoch   8 Step:     2700 Batch Loss:     3.314483 Tokens per Sec:     7133, Lr: 0.000300\n",
            "2019-11-26 15:58:51,938 Epoch   8 Step:     2800 Batch Loss:     3.295641 Tokens per Sec:     7229, Lr: 0.000300\n",
            "2019-11-26 15:59:23,408 Epoch   8 Step:     2900 Batch Loss:     3.611770 Tokens per Sec:     6958, Lr: 0.000300\n",
            "2019-11-26 15:59:54,275 Epoch   8: total training loss 1228.62\n",
            "2019-11-26 15:59:54,275 EPOCH 9\n",
            "2019-11-26 15:59:54,633 Epoch   9 Step:     3000 Batch Loss:     3.129238 Tokens per Sec:     6813, Lr: 0.000300\n",
            "2019-11-26 16:00:25,840 Epoch   9 Step:     3100 Batch Loss:     3.271988 Tokens per Sec:     7056, Lr: 0.000300\n",
            "2019-11-26 16:00:57,494 Epoch   9 Step:     3200 Batch Loss:     3.338350 Tokens per Sec:     7186, Lr: 0.000300\n",
            "2019-11-26 16:01:28,389 Epoch   9 Step:     3300 Batch Loss:     2.963907 Tokens per Sec:     6907, Lr: 0.000300\n",
            "2019-11-26 16:01:51,645 Epoch   9: total training loss 1198.45\n",
            "2019-11-26 16:01:51,645 EPOCH 10\n",
            "2019-11-26 16:01:59,544 Epoch  10 Step:     3400 Batch Loss:     2.930466 Tokens per Sec:     7099, Lr: 0.000300\n",
            "2019-11-26 16:02:30,961 Epoch  10 Step:     3500 Batch Loss:     2.998262 Tokens per Sec:     7204, Lr: 0.000300\n",
            "2019-11-26 16:03:02,425 Epoch  10 Step:     3600 Batch Loss:     2.363204 Tokens per Sec:     7098, Lr: 0.000300\n",
            "2019-11-26 16:03:33,643 Epoch  10 Step:     3700 Batch Loss:     2.931180 Tokens per Sec:     6919, Lr: 0.000300\n",
            "2019-11-26 16:03:48,857 Epoch  10: total training loss 1160.02\n",
            "2019-11-26 16:03:48,857 EPOCH 11\n",
            "2019-11-26 16:04:04,869 Epoch  11 Step:     3800 Batch Loss:     3.319844 Tokens per Sec:     7062, Lr: 0.000300\n",
            "2019-11-26 16:04:35,827 Epoch  11 Step:     3900 Batch Loss:     3.233286 Tokens per Sec:     7013, Lr: 0.000300\n",
            "2019-11-26 16:05:07,185 Epoch  11 Step:     4000 Batch Loss:     3.136714 Tokens per Sec:     7207, Lr: 0.000300\n",
            "2019-11-26 16:06:47,854 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 16:06:47,854 Saving new checkpoint.\n",
            "2019-11-26 16:06:48,116 Example #0\n",
            "2019-11-26 16:06:48,117 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 16:06:48,117 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 16:06:48,117 \tHypothesis: Mošomo wa go feta o tla ba le maikarabelo a go ba le maikarabelo a go ba le batho ba rena ba rena ba rena.\n",
            "2019-11-26 16:06:48,118 Example #1\n",
            "2019-11-26 16:06:48,118 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 16:06:48,118 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 16:06:48,118 \tHypothesis: Dipoledišano tše di lego gona di šoma ka go ba le go ba le go ba le go ba le go feta go tloga go tloga go tloga go tloga go fihla ka nako ye e fetilego ka go tloga go fihla ka Junibesena.\n",
            "2019-11-26 16:06:48,118 Example #2\n",
            "2019-11-26 16:06:48,118 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:06:48,119 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:06:48,119 \tHypothesis: Kgopelo ya go ngwadiša khamphani ya Afrika Borwa ya Afrika Borwa\n",
            "2019-11-26 16:06:48,119 Example #3\n",
            "2019-11-26 16:06:48,119 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 16:06:48,119 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 16:06:48,119 \tHypothesis: Mmušo o tla ba le tshedimošo ka moka ge e le gore o na le maikarabelo a go dira gore o kgone go ba le go ba le go ba le go ba le go se be go se na le go se na le go se na le go se be go se be go se na le go se be go se be go se be go se na le go se be go se se se se na le go se se se se se se se se se se se se se se se se se\n",
            "2019-11-26 16:06:48,119 Validation result (greedy) at epoch  11, step     4000: bleu:   2.03, loss: 99850.7344, ppl:  20.2892, duration: 100.9340s\n",
            "2019-11-26 16:07:19,400 Epoch  11 Step:     4100 Batch Loss:     3.104984 Tokens per Sec:     7067, Lr: 0.000300\n",
            "2019-11-26 16:07:26,671 Epoch  11: total training loss 1130.84\n",
            "2019-11-26 16:07:26,672 EPOCH 12\n",
            "2019-11-26 16:07:51,071 Epoch  12 Step:     4200 Batch Loss:     2.931280 Tokens per Sec:     7346, Lr: 0.000300\n",
            "2019-11-26 16:08:21,928 Epoch  12 Step:     4300 Batch Loss:     2.980639 Tokens per Sec:     7020, Lr: 0.000300\n",
            "2019-11-26 16:08:52,923 Epoch  12 Step:     4400 Batch Loss:     3.052874 Tokens per Sec:     7091, Lr: 0.000300\n",
            "2019-11-26 16:09:22,920 Epoch  12: total training loss 1103.24\n",
            "2019-11-26 16:09:22,920 EPOCH 13\n",
            "2019-11-26 16:09:23,967 Epoch  13 Step:     4500 Batch Loss:     2.951346 Tokens per Sec:     6612, Lr: 0.000300\n",
            "2019-11-26 16:09:55,172 Epoch  13 Step:     4600 Batch Loss:     3.138725 Tokens per Sec:     7187, Lr: 0.000300\n",
            "2019-11-26 16:10:26,629 Epoch  13 Step:     4700 Batch Loss:     3.209488 Tokens per Sec:     7170, Lr: 0.000300\n",
            "2019-11-26 16:10:57,129 Epoch  13 Step:     4800 Batch Loss:     2.233067 Tokens per Sec:     6972, Lr: 0.000300\n",
            "2019-11-26 16:11:19,433 Epoch  13: total training loss 1082.63\n",
            "2019-11-26 16:11:19,434 EPOCH 14\n",
            "2019-11-26 16:11:28,260 Epoch  14 Step:     4900 Batch Loss:     3.030929 Tokens per Sec:     6837, Lr: 0.000300\n",
            "2019-11-26 16:11:59,330 Epoch  14 Step:     5000 Batch Loss:     2.898174 Tokens per Sec:     7073, Lr: 0.000300\n",
            "2019-11-26 16:12:30,477 Epoch  14 Step:     5100 Batch Loss:     2.938483 Tokens per Sec:     7126, Lr: 0.000300\n",
            "2019-11-26 16:13:01,659 Epoch  14 Step:     5200 Batch Loss:     2.238496 Tokens per Sec:     7157, Lr: 0.000300\n",
            "2019-11-26 16:13:15,926 Epoch  14: total training loss 1060.73\n",
            "2019-11-26 16:13:15,926 EPOCH 15\n",
            "2019-11-26 16:13:32,621 Epoch  15 Step:     5300 Batch Loss:     2.768561 Tokens per Sec:     6967, Lr: 0.000300\n",
            "2019-11-26 16:14:03,877 Epoch  15 Step:     5400 Batch Loss:     3.139548 Tokens per Sec:     7087, Lr: 0.000300\n",
            "2019-11-26 16:14:35,203 Epoch  15 Step:     5500 Batch Loss:     2.992505 Tokens per Sec:     7177, Lr: 0.000300\n",
            "2019-11-26 16:15:06,577 Epoch  15 Step:     5600 Batch Loss:     2.912321 Tokens per Sec:     7145, Lr: 0.000300\n",
            "2019-11-26 16:15:12,580 Epoch  15: total training loss 1032.01\n",
            "2019-11-26 16:15:12,580 EPOCH 16\n",
            "2019-11-26 16:15:38,123 Epoch  16 Step:     5700 Batch Loss:     2.908057 Tokens per Sec:     7147, Lr: 0.000300\n",
            "2019-11-26 16:16:09,652 Epoch  16 Step:     5800 Batch Loss:     3.057437 Tokens per Sec:     7126, Lr: 0.000300\n",
            "2019-11-26 16:16:41,080 Epoch  16 Step:     5900 Batch Loss:     2.844653 Tokens per Sec:     7054, Lr: 0.000300\n",
            "2019-11-26 16:17:09,476 Epoch  16: total training loss 1014.14\n",
            "2019-11-26 16:17:09,476 EPOCH 17\n",
            "2019-11-26 16:17:12,000 Epoch  17 Step:     6000 Batch Loss:     2.487259 Tokens per Sec:     7069, Lr: 0.000300\n",
            "2019-11-26 16:18:51,790 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 16:18:51,790 Saving new checkpoint.\n",
            "2019-11-26 16:18:52,058 Example #0\n",
            "2019-11-26 16:18:52,059 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 16:18:52,059 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 16:18:52,059 \tHypothesis: Leano la bosenyi le le le letelele la go aga la kgethologanyo e tla ba le seabe sa go feta ka moka ka moka ga batho ba rena.\n",
            "2019-11-26 16:18:52,059 Example #1\n",
            "2019-11-26 16:18:52,059 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 16:18:52,059 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 16:18:52,060 \tHypothesis: Dipoelo tša go ikgetha di ile tša dirwa ka go ba le seabe sa go se lekalekane le go feta go fihla go fihla go fihla go tše tharo tša ka Julay go fihla go fihla go fihla go tše nne tše di sa tšwago go feta go feta go feta go ya go feta go feta go ya go feta .\n",
            "2019-11-26 16:18:52,060 Example #2\n",
            "2019-11-26 16:18:52,060 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:18:52,060 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:18:52,060 \tHypothesis: Kgopelo ya go fetola boagi bja Afrika Borwa bja Afrika Borwa\n",
            "2019-11-26 16:18:52,060 Example #3\n",
            "2019-11-26 16:18:52,061 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 16:18:52,061 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 16:18:52,061 \tHypothesis: Tshedimošo ya rena e fa tshedimošo ka ge e le gore ga e na le tshedimošo ye e sa nyakegago.\n",
            "2019-11-26 16:18:52,061 Validation result (greedy) at epoch  17, step     6000: bleu:   4.43, loss: 90909.4375, ppl:  15.4955, duration: 100.0609s\n",
            "2019-11-26 16:19:23,437 Epoch  17 Step:     6100 Batch Loss:     2.610669 Tokens per Sec:     7218, Lr: 0.000300\n",
            "2019-11-26 16:19:54,270 Epoch  17 Step:     6200 Batch Loss:     2.747778 Tokens per Sec:     7064, Lr: 0.000300\n",
            "2019-11-26 16:20:25,001 Epoch  17 Step:     6300 Batch Loss:     2.443906 Tokens per Sec:     7091, Lr: 0.000300\n",
            "2019-11-26 16:20:46,245 Epoch  17: total training loss 1004.02\n",
            "2019-11-26 16:20:46,246 EPOCH 18\n",
            "2019-11-26 16:20:56,158 Epoch  18 Step:     6400 Batch Loss:     2.518567 Tokens per Sec:     7316, Lr: 0.000300\n",
            "2019-11-26 16:21:27,282 Epoch  18 Step:     6500 Batch Loss:     1.761815 Tokens per Sec:     7122, Lr: 0.000300\n",
            "2019-11-26 16:21:58,110 Epoch  18 Step:     6600 Batch Loss:     2.884588 Tokens per Sec:     6959, Lr: 0.000300\n",
            "2019-11-26 16:22:28,888 Epoch  18 Step:     6700 Batch Loss:     2.598293 Tokens per Sec:     7056, Lr: 0.000300\n",
            "2019-11-26 16:22:43,475 Epoch  18: total training loss 987.27\n",
            "2019-11-26 16:22:43,476 EPOCH 19\n",
            "2019-11-26 16:23:00,052 Epoch  19 Step:     6800 Batch Loss:     2.676794 Tokens per Sec:     7155, Lr: 0.000300\n",
            "2019-11-26 16:23:31,520 Epoch  19 Step:     6900 Batch Loss:     2.517348 Tokens per Sec:     7220, Lr: 0.000300\n",
            "2019-11-26 16:24:02,947 Epoch  19 Step:     7000 Batch Loss:     2.382189 Tokens per Sec:     7121, Lr: 0.000300\n",
            "2019-11-26 16:24:33,629 Epoch  19 Step:     7100 Batch Loss:     2.645994 Tokens per Sec:     6785, Lr: 0.000300\n",
            "2019-11-26 16:24:40,514 Epoch  19: total training loss 963.67\n",
            "2019-11-26 16:24:40,514 EPOCH 20\n",
            "2019-11-26 16:25:04,954 Epoch  20 Step:     7200 Batch Loss:     2.442765 Tokens per Sec:     7157, Lr: 0.000300\n",
            "2019-11-26 16:25:36,088 Epoch  20 Step:     7300 Batch Loss:     2.511607 Tokens per Sec:     7188, Lr: 0.000300\n",
            "2019-11-26 16:26:07,153 Epoch  20 Step:     7400 Batch Loss:     2.692013 Tokens per Sec:     7130, Lr: 0.000300\n",
            "2019-11-26 16:26:36,489 Epoch  20: total training loss 943.99\n",
            "2019-11-26 16:26:36,490 EPOCH 21\n",
            "2019-11-26 16:26:37,979 Epoch  21 Step:     7500 Batch Loss:     1.568056 Tokens per Sec:     5278, Lr: 0.000300\n",
            "2019-11-26 16:27:08,776 Epoch  21 Step:     7600 Batch Loss:     2.141893 Tokens per Sec:     7060, Lr: 0.000300\n",
            "2019-11-26 16:27:40,027 Epoch  21 Step:     7700 Batch Loss:     2.551643 Tokens per Sec:     7148, Lr: 0.000300\n",
            "2019-11-26 16:28:11,441 Epoch  21 Step:     7800 Batch Loss:     1.632171 Tokens per Sec:     7159, Lr: 0.000300\n",
            "2019-11-26 16:28:33,514 Epoch  21: total training loss 936.40\n",
            "2019-11-26 16:28:33,514 EPOCH 22\n",
            "2019-11-26 16:28:42,606 Epoch  22 Step:     7900 Batch Loss:     2.243351 Tokens per Sec:     7069, Lr: 0.000300\n",
            "2019-11-26 16:29:13,854 Epoch  22 Step:     8000 Batch Loss:     1.458223 Tokens per Sec:     7030, Lr: 0.000300\n",
            "2019-11-26 16:30:53,926 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 16:30:53,926 Saving new checkpoint.\n",
            "2019-11-26 16:30:54,228 Example #0\n",
            "2019-11-26 16:30:54,228 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 16:30:54,229 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 16:30:54,229 \tHypothesis: Bosenyi kgahlanong le bosenyi bo a nyakega gore tshepedišo ya kgethologanyo e tla ba le seabe ka moka ya batho ka moka ba bannyane ka nageng.\n",
            "2019-11-26 16:30:54,229 Example #1\n",
            "2019-11-26 16:30:54,229 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 16:30:54,229 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 16:30:54,229 \tHypothesis: Kabinete le yona e tsebagaditše go ba le go se be le go se na le seabe sa lefase kgahlanong le dinaga tše mmalwa tše di fetilego tše di fetilego tše di fetilego tše di fetilego .\n",
            "2019-11-26 16:30:54,229 Example #2\n",
            "2019-11-26 16:30:54,230 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:30:54,230 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:30:54,230 \tHypothesis: Kgopelo ya tumelelo ya bodudi bja Afrika Borwa bja Afrika Borwa\n",
            "2019-11-26 16:30:54,230 Example #3\n",
            "2019-11-26 16:30:54,230 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 16:30:54,230 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 16:30:54,231 \tHypothesis: Tshedimošo ya rena ke ya tshedimošo yeo e bego e le gore e be e le gona.\n",
            "2019-11-26 16:30:54,231 Validation result (greedy) at epoch  22, step     8000: bleu:   7.01, loss: 85468.1562, ppl:  13.1512, duration: 100.3765s\n",
            "2019-11-26 16:31:25,621 Epoch  22 Step:     8100 Batch Loss:     2.350302 Tokens per Sec:     7142, Lr: 0.000300\n",
            "2019-11-26 16:31:56,720 Epoch  22 Step:     8200 Batch Loss:     2.406871 Tokens per Sec:     7105, Lr: 0.000300\n",
            "2019-11-26 16:32:10,577 Epoch  22: total training loss 917.35\n",
            "2019-11-26 16:32:10,577 EPOCH 23\n",
            "2019-11-26 16:32:27,699 Epoch  23 Step:     8300 Batch Loss:     1.661386 Tokens per Sec:     7090, Lr: 0.000300\n",
            "2019-11-26 16:32:59,037 Epoch  23 Step:     8400 Batch Loss:     2.362143 Tokens per Sec:     7244, Lr: 0.000300\n",
            "2019-11-26 16:33:29,992 Epoch  23 Step:     8500 Batch Loss:     2.585480 Tokens per Sec:     7019, Lr: 0.000300\n",
            "2019-11-26 16:34:01,384 Epoch  23 Step:     8600 Batch Loss:     2.744380 Tokens per Sec:     7224, Lr: 0.000300\n",
            "2019-11-26 16:34:06,801 Epoch  23: total training loss 900.56\n",
            "2019-11-26 16:34:06,801 EPOCH 24\n",
            "2019-11-26 16:34:32,401 Epoch  24 Step:     8700 Batch Loss:     1.502004 Tokens per Sec:     7026, Lr: 0.000300\n",
            "2019-11-26 16:35:03,297 Epoch  24 Step:     8800 Batch Loss:     1.422535 Tokens per Sec:     7011, Lr: 0.000300\n",
            "2019-11-26 16:35:34,522 Epoch  24 Step:     8900 Batch Loss:     2.205901 Tokens per Sec:     7113, Lr: 0.000300\n",
            "2019-11-26 16:36:03,821 Epoch  24: total training loss 892.59\n",
            "2019-11-26 16:36:03,821 EPOCH 25\n",
            "2019-11-26 16:36:05,778 Epoch  25 Step:     9000 Batch Loss:     2.427428 Tokens per Sec:     7203, Lr: 0.000300\n",
            "2019-11-26 16:36:36,523 Epoch  25 Step:     9100 Batch Loss:     2.292642 Tokens per Sec:     6902, Lr: 0.000300\n",
            "2019-11-26 16:37:07,958 Epoch  25 Step:     9200 Batch Loss:     1.556768 Tokens per Sec:     7117, Lr: 0.000300\n",
            "2019-11-26 16:37:39,248 Epoch  25 Step:     9300 Batch Loss:     2.616799 Tokens per Sec:     7200, Lr: 0.000300\n",
            "2019-11-26 16:38:01,047 Epoch  25: total training loss 880.17\n",
            "2019-11-26 16:38:01,048 EPOCH 26\n",
            "2019-11-26 16:38:10,262 Epoch  26 Step:     9400 Batch Loss:     2.426787 Tokens per Sec:     6952, Lr: 0.000300\n",
            "2019-11-26 16:38:41,706 Epoch  26 Step:     9500 Batch Loss:     2.310549 Tokens per Sec:     7024, Lr: 0.000300\n",
            "2019-11-26 16:39:13,144 Epoch  26 Step:     9600 Batch Loss:     2.668702 Tokens per Sec:     7071, Lr: 0.000300\n",
            "2019-11-26 16:39:44,414 Epoch  26 Step:     9700 Batch Loss:     2.451712 Tokens per Sec:     7187, Lr: 0.000300\n",
            "2019-11-26 16:39:57,985 Epoch  26: total training loss 863.42\n",
            "2019-11-26 16:39:57,985 EPOCH 27\n",
            "2019-11-26 16:40:15,532 Epoch  27 Step:     9800 Batch Loss:     2.138623 Tokens per Sec:     6995, Lr: 0.000300\n",
            "2019-11-26 16:40:46,658 Epoch  27 Step:     9900 Batch Loss:     2.627006 Tokens per Sec:     7109, Lr: 0.000300\n",
            "2019-11-26 16:41:18,070 Epoch  27 Step:    10000 Batch Loss:     2.437124 Tokens per Sec:     7142, Lr: 0.000300\n",
            "2019-11-26 16:42:57,877 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 16:42:57,877 Saving new checkpoint.\n",
            "2019-11-26 16:42:58,174 Example #0\n",
            "2019-11-26 16:42:58,174 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 16:42:58,174 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 16:42:58,176 \tHypothesis: Bosenyi bja bosenyi bo a le gore tshepedišo ya kgethologanyo e tla ba le seabe sa go ba le seabe ka nageng ka moka.\n",
            "2019-11-26 16:42:58,176 Example #1\n",
            "2019-11-26 16:42:58,177 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 16:42:58,177 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 16:42:58,177 \tHypothesis: Kabinete e ile ya fetošwa bjalo ka ge e le gore e be e le gore e be e le gore e be e le ka Julae mo dibekeng tše lesomepedi.\n",
            "2019-11-26 16:42:58,177 Example #2\n",
            "2019-11-26 16:42:58,177 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:42:58,177 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:42:58,177 \tHypothesis: Kgopelo ya go fetola boagi bja Afrika Borwa, 75\n",
            "2019-11-26 16:42:58,178 Example #3\n",
            "2019-11-26 16:42:58,178 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 16:42:58,178 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 16:42:58,178 \tHypothesis: Molaotheo wa rena o fana ka tshedimošo ge e le gore e ba e na le dikgato tše di sa nyakegego go tšona.\n",
            "2019-11-26 16:42:58,178 Validation result (greedy) at epoch  27, step    10000: bleu:   8.59, loss: 81320.2500, ppl:  11.6054, duration: 100.1076s\n",
            "2019-11-26 16:43:29,236 Epoch  27 Step:    10100 Batch Loss:     2.490533 Tokens per Sec:     7058, Lr: 0.000300\n",
            "2019-11-26 16:43:34,823 Epoch  27: total training loss 852.96\n",
            "2019-11-26 16:43:34,823 EPOCH 28\n",
            "2019-11-26 16:44:00,533 Epoch  28 Step:    10200 Batch Loss:     2.304375 Tokens per Sec:     7242, Lr: 0.000300\n",
            "2019-11-26 16:44:31,249 Epoch  28 Step:    10300 Batch Loss:     2.148170 Tokens per Sec:     6813, Lr: 0.000300\n",
            "2019-11-26 16:45:02,608 Epoch  28 Step:    10400 Batch Loss:     2.197670 Tokens per Sec:     7138, Lr: 0.000300\n",
            "2019-11-26 16:45:32,084 Epoch  28: total training loss 844.81\n",
            "2019-11-26 16:45:32,084 EPOCH 29\n",
            "2019-11-26 16:45:34,076 Epoch  29 Step:    10500 Batch Loss:     2.449766 Tokens per Sec:     7238, Lr: 0.000300\n",
            "2019-11-26 16:46:05,341 Epoch  29 Step:    10600 Batch Loss:     2.598052 Tokens per Sec:     7005, Lr: 0.000300\n",
            "2019-11-26 16:46:36,568 Epoch  29 Step:    10700 Batch Loss:     2.402862 Tokens per Sec:     7218, Lr: 0.000300\n",
            "2019-11-26 16:47:07,640 Epoch  29 Step:    10800 Batch Loss:     2.286349 Tokens per Sec:     7009, Lr: 0.000300\n",
            "2019-11-26 16:47:28,887 Epoch  29: total training loss 831.31\n",
            "2019-11-26 16:47:28,888 EPOCH 30\n",
            "2019-11-26 16:47:38,820 Epoch  30 Step:    10900 Batch Loss:     2.119458 Tokens per Sec:     7274, Lr: 0.000300\n",
            "2019-11-26 16:48:09,844 Epoch  30 Step:    11000 Batch Loss:     2.387829 Tokens per Sec:     7112, Lr: 0.000300\n",
            "2019-11-26 16:48:41,133 Epoch  30 Step:    11100 Batch Loss:     1.888255 Tokens per Sec:     7075, Lr: 0.000300\n",
            "2019-11-26 16:49:11,916 Epoch  30 Step:    11200 Batch Loss:     2.235555 Tokens per Sec:     7109, Lr: 0.000300\n",
            "2019-11-26 16:49:25,698 Epoch  30: total training loss 823.94\n",
            "2019-11-26 16:49:25,698 EPOCH 31\n",
            "2019-11-26 16:49:42,880 Epoch  31 Step:    11300 Batch Loss:     2.415681 Tokens per Sec:     7227, Lr: 0.000300\n",
            "2019-11-26 16:50:14,406 Epoch  31 Step:    11400 Batch Loss:     1.317502 Tokens per Sec:     7130, Lr: 0.000300\n",
            "2019-11-26 16:50:45,319 Epoch  31 Step:    11500 Batch Loss:     1.202223 Tokens per Sec:     7005, Lr: 0.000300\n",
            "2019-11-26 16:51:16,545 Epoch  31 Step:    11600 Batch Loss:     2.134407 Tokens per Sec:     7055, Lr: 0.000300\n",
            "2019-11-26 16:51:22,419 Epoch  31: total training loss 810.76\n",
            "2019-11-26 16:51:22,420 EPOCH 32\n",
            "2019-11-26 16:51:47,621 Epoch  32 Step:    11700 Batch Loss:     2.411912 Tokens per Sec:     7027, Lr: 0.000300\n",
            "2019-11-26 16:52:18,932 Epoch  32 Step:    11800 Batch Loss:     2.249519 Tokens per Sec:     7223, Lr: 0.000300\n",
            "2019-11-26 16:52:50,149 Epoch  32 Step:    11900 Batch Loss:     2.160145 Tokens per Sec:     7187, Lr: 0.000300\n",
            "2019-11-26 16:53:18,921 Epoch  32: total training loss 798.67\n",
            "2019-11-26 16:53:18,921 EPOCH 33\n",
            "2019-11-26 16:53:21,189 Epoch  33 Step:    12000 Batch Loss:     2.224640 Tokens per Sec:     6678, Lr: 0.000300\n",
            "2019-11-26 16:55:00,832 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 16:55:00,832 Saving new checkpoint.\n",
            "2019-11-26 16:55:01,121 Example #0\n",
            "2019-11-26 16:55:01,122 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 16:55:01,122 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 16:55:01,123 \tHypothesis: Bosenyi bja bosenyi bo a tseba, ke gore tshepedišo ya kgethologanyo e tla ba le seabe ka moka ka nageng ya rena.\n",
            "2019-11-26 16:55:01,123 Example #1\n",
            "2019-11-26 16:55:01,123 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 16:55:01,123 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 16:55:01,123 \tHypothesis: Kabinete e ile ya fetošwa ka ge e le gore e be e le gore e be e le mogopolo wa go feta mo dibekeng tše seswai tša go feta tše di fetilego tše di fetilego tše di fetilego .\n",
            "2019-11-26 16:55:01,123 Example #2\n",
            "2019-11-26 16:55:01,124 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:55:01,124 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 16:55:01,124 \tHypothesis: Kgopelo ya go gogela boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 16:55:01,124 Example #3\n",
            "2019-11-26 16:55:01,124 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 16:55:01,124 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 16:55:01,124 \tHypothesis: Molaotshepetšo wa rena o fana ka tshedimošo gomme ge e le gore e na le dikgato tše di lego gona ka tsela ye e lego gona.\n",
            "2019-11-26 16:55:01,124 Validation result (greedy) at epoch  33, step    12000: bleu:  10.28, loss: 78477.8672, ppl:  10.6524, duration: 99.9352s\n",
            "2019-11-26 16:55:32,701 Epoch  33 Step:    12100 Batch Loss:     2.449997 Tokens per Sec:     7235, Lr: 0.000300\n",
            "2019-11-26 16:56:03,983 Epoch  33 Step:    12200 Batch Loss:     2.086523 Tokens per Sec:     7047, Lr: 0.000300\n",
            "2019-11-26 16:56:35,129 Epoch  33 Step:    12300 Batch Loss:     2.063632 Tokens per Sec:     7093, Lr: 0.000300\n",
            "2019-11-26 16:56:55,650 Epoch  33: total training loss 787.84\n",
            "2019-11-26 16:56:55,650 EPOCH 34\n",
            "2019-11-26 16:57:06,247 Epoch  34 Step:    12400 Batch Loss:     2.068075 Tokens per Sec:     7000, Lr: 0.000300\n",
            "2019-11-26 16:57:37,612 Epoch  34 Step:    12500 Batch Loss:     2.353165 Tokens per Sec:     7179, Lr: 0.000300\n",
            "2019-11-26 16:58:08,770 Epoch  34 Step:    12600 Batch Loss:     2.329750 Tokens per Sec:     7026, Lr: 0.000300\n",
            "2019-11-26 16:58:39,693 Epoch  34 Step:    12700 Batch Loss:     2.083556 Tokens per Sec:     7091, Lr: 0.000300\n",
            "2019-11-26 16:58:52,431 Epoch  34: total training loss 781.28\n",
            "2019-11-26 16:58:52,432 EPOCH 35\n",
            "2019-11-26 16:59:10,651 Epoch  35 Step:    12800 Batch Loss:     2.283317 Tokens per Sec:     6984, Lr: 0.000300\n",
            "2019-11-26 16:59:42,086 Epoch  35 Step:    12900 Batch Loss:     2.078687 Tokens per Sec:     7211, Lr: 0.000300\n",
            "2019-11-26 17:00:13,822 Epoch  35 Step:    13000 Batch Loss:     1.900955 Tokens per Sec:     7170, Lr: 0.000300\n",
            "2019-11-26 17:00:44,824 Epoch  35 Step:    13100 Batch Loss:     2.460504 Tokens per Sec:     7038, Lr: 0.000300\n",
            "2019-11-26 17:00:48,993 Epoch  35: total training loss 771.40\n",
            "2019-11-26 17:00:48,993 EPOCH 36\n",
            "2019-11-26 17:01:16,215 Epoch  36 Step:    13200 Batch Loss:     2.231016 Tokens per Sec:     7214, Lr: 0.000300\n",
            "2019-11-26 17:01:47,001 Epoch  36 Step:    13300 Batch Loss:     1.907635 Tokens per Sec:     7009, Lr: 0.000300\n",
            "2019-11-26 17:02:18,463 Epoch  36 Step:    13400 Batch Loss:     1.122081 Tokens per Sec:     7169, Lr: 0.000300\n",
            "2019-11-26 17:02:45,697 Epoch  36: total training loss 761.03\n",
            "2019-11-26 17:02:45,697 EPOCH 37\n",
            "2019-11-26 17:02:49,564 Epoch  37 Step:    13500 Batch Loss:     2.366977 Tokens per Sec:     6974, Lr: 0.000300\n",
            "2019-11-26 17:03:20,983 Epoch  37 Step:    13600 Batch Loss:     1.722432 Tokens per Sec:     7216, Lr: 0.000300\n",
            "2019-11-26 17:03:51,984 Epoch  37 Step:    13700 Batch Loss:     2.176898 Tokens per Sec:     7059, Lr: 0.000300\n",
            "2019-11-26 17:04:23,365 Epoch  37 Step:    13800 Batch Loss:     1.949642 Tokens per Sec:     7133, Lr: 0.000300\n",
            "2019-11-26 17:04:42,096 Epoch  37: total training loss 750.47\n",
            "2019-11-26 17:04:42,096 EPOCH 38\n",
            "2019-11-26 17:04:54,598 Epoch  38 Step:    13900 Batch Loss:     1.515741 Tokens per Sec:     7003, Lr: 0.000300\n",
            "2019-11-26 17:05:25,967 Epoch  38 Step:    14000 Batch Loss:     1.824513 Tokens per Sec:     7168, Lr: 0.000300\n",
            "2019-11-26 17:07:05,665 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 17:07:05,666 Saving new checkpoint.\n",
            "2019-11-26 17:07:05,962 Example #0\n",
            "2019-11-26 17:07:05,963 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 17:07:05,963 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 17:07:05,963 \tHypothesis: Bosenyi bja bosenyi bo a hlokega gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ba bathobaso ka moka ba baso ka nageng ya rena.\n",
            "2019-11-26 17:07:05,963 Example #1\n",
            "2019-11-26 17:07:05,964 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 17:07:05,964 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 17:07:05,964 \tHypothesis: Kabinete gape e ile ya feletša ka go šomiša tshedimošo ya go swana le dikgopolo tša go feta.\n",
            "2019-11-26 17:07:05,964 Example #2\n",
            "2019-11-26 17:07:05,964 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:07:05,965 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:07:05,965 \tHypothesis: Kgopelo ya go gogela boagi bja Afrika Borwa, 75\n",
            "2019-11-26 17:07:05,965 Example #3\n",
            "2019-11-26 17:07:05,965 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 17:07:05,965 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 17:07:05,965 \tHypothesis: Molaotshepetšo wa rena o fana ka tshedimošo ge e ba e ba gona e ba e na le dikgopolo tše di lego gona.\n",
            "2019-11-26 17:07:05,965 Validation result (greedy) at epoch  38, step    14000: bleu:  11.76, loss: 76418.5938, ppl:  10.0112, duration: 99.9982s\n",
            "2019-11-26 17:07:37,175 Epoch  38 Step:    14100 Batch Loss:     2.432888 Tokens per Sec:     7191, Lr: 0.000300\n",
            "2019-11-26 17:08:07,958 Epoch  38 Step:    14200 Batch Loss:     2.117253 Tokens per Sec:     6950, Lr: 0.000300\n",
            "2019-11-26 17:08:18,668 Epoch  38: total training loss 746.33\n",
            "2019-11-26 17:08:18,668 EPOCH 39\n",
            "2019-11-26 17:08:39,215 Epoch  39 Step:    14300 Batch Loss:     2.273044 Tokens per Sec:     7077, Lr: 0.000300\n",
            "2019-11-26 17:09:10,667 Epoch  39 Step:    14400 Batch Loss:     2.411302 Tokens per Sec:     7065, Lr: 0.000300\n",
            "2019-11-26 17:09:41,862 Epoch  39 Step:    14500 Batch Loss:     1.942470 Tokens per Sec:     7128, Lr: 0.000300\n",
            "2019-11-26 17:10:13,161 Epoch  39 Step:    14600 Batch Loss:     2.155474 Tokens per Sec:     7158, Lr: 0.000300\n",
            "2019-11-26 17:10:15,384 Epoch  39: total training loss 738.26\n",
            "2019-11-26 17:10:15,384 EPOCH 40\n",
            "2019-11-26 17:10:44,468 Epoch  40 Step:    14700 Batch Loss:     1.300541 Tokens per Sec:     6918, Lr: 0.000300\n",
            "2019-11-26 17:11:15,813 Epoch  40 Step:    14800 Batch Loss:     2.288501 Tokens per Sec:     7159, Lr: 0.000300\n",
            "2019-11-26 17:11:46,709 Epoch  40 Step:    14900 Batch Loss:     2.115719 Tokens per Sec:     7123, Lr: 0.000300\n",
            "2019-11-26 17:12:12,362 Epoch  40: total training loss 734.77\n",
            "2019-11-26 17:12:12,362 EPOCH 41\n",
            "2019-11-26 17:12:18,143 Epoch  41 Step:    15000 Batch Loss:     2.222058 Tokens per Sec:     7177, Lr: 0.000300\n",
            "2019-11-26 17:12:49,149 Epoch  41 Step:    15100 Batch Loss:     2.221619 Tokens per Sec:     7042, Lr: 0.000300\n",
            "2019-11-26 17:13:20,249 Epoch  41 Step:    15200 Batch Loss:     2.287112 Tokens per Sec:     7117, Lr: 0.000300\n",
            "2019-11-26 17:13:51,285 Epoch  41 Step:    15300 Batch Loss:     2.248198 Tokens per Sec:     7061, Lr: 0.000300\n",
            "2019-11-26 17:14:09,273 Epoch  41: total training loss 729.25\n",
            "2019-11-26 17:14:09,273 EPOCH 42\n",
            "2019-11-26 17:14:22,158 Epoch  42 Step:    15400 Batch Loss:     1.895740 Tokens per Sec:     7137, Lr: 0.000300\n",
            "2019-11-26 17:14:53,498 Epoch  42 Step:    15500 Batch Loss:     2.010869 Tokens per Sec:     7028, Lr: 0.000300\n",
            "2019-11-26 17:15:24,937 Epoch  42 Step:    15600 Batch Loss:     2.221094 Tokens per Sec:     7227, Lr: 0.000300\n",
            "2019-11-26 17:15:56,139 Epoch  42 Step:    15700 Batch Loss:     1.058546 Tokens per Sec:     7111, Lr: 0.000300\n",
            "2019-11-26 17:16:05,936 Epoch  42: total training loss 715.21\n",
            "2019-11-26 17:16:05,936 EPOCH 43\n",
            "2019-11-26 17:16:27,374 Epoch  43 Step:    15800 Batch Loss:     1.895528 Tokens per Sec:     7216, Lr: 0.000300\n",
            "2019-11-26 17:16:58,561 Epoch  43 Step:    15900 Batch Loss:     2.170236 Tokens per Sec:     7160, Lr: 0.000300\n",
            "2019-11-26 17:17:29,953 Epoch  43 Step:    16000 Batch Loss:     1.930803 Tokens per Sec:     7041, Lr: 0.000300\n",
            "2019-11-26 17:19:09,468 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 17:19:09,468 Saving new checkpoint.\n",
            "2019-11-26 17:19:09,772 Example #0\n",
            "2019-11-26 17:19:09,772 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 17:19:09,772 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 17:19:09,772 \tHypothesis: Bosenyi kgahlanong le botho bja go se lekalekane gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ba baso ka moka ka nageng ya rena.\n",
            "2019-11-26 17:19:09,773 Example #1\n",
            "2019-11-26 17:19:09,773 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 17:19:09,773 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 17:19:09,773 \tHypothesis: Kabinete le yona e ile ya fetošwa ka ge e le gore e be e le gore e se ke ya ba ye e bego e le ye e fetilego.\n",
            "2019-11-26 17:19:09,773 Example #2\n",
            "2019-11-26 17:19:09,774 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:19:09,774 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:19:09,774 \tHypothesis: Application for retenation of South African citizenship, BI-175\n",
            "2019-11-26 17:19:09,774 Example #3\n",
            "2019-11-26 17:19:09,774 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 17:19:09,774 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 17:19:09,774 \tHypothesis: Molaotshepetšo wa rena o fana ka tshedimošo ge e le gore e ba gona e ba e le gona.\n",
            "2019-11-26 17:19:09,775 Validation result (greedy) at epoch  43, step    16000: bleu:  13.04, loss: 74479.3359, ppl:   9.4427, duration: 99.8213s\n",
            "2019-11-26 17:19:40,752 Epoch  43 Step:    16100 Batch Loss:     2.138124 Tokens per Sec:     7052, Lr: 0.000300\n",
            "2019-11-26 17:19:42,463 Epoch  43: total training loss 712.30\n",
            "2019-11-26 17:19:42,463 EPOCH 44\n",
            "2019-11-26 17:20:11,894 Epoch  44 Step:    16200 Batch Loss:     1.916833 Tokens per Sec:     7032, Lr: 0.000300\n",
            "2019-11-26 17:20:43,076 Epoch  44 Step:    16300 Batch Loss:     1.899516 Tokens per Sec:     7192, Lr: 0.000300\n",
            "2019-11-26 17:21:14,368 Epoch  44 Step:    16400 Batch Loss:     2.124926 Tokens per Sec:     7124, Lr: 0.000300\n",
            "2019-11-26 17:21:39,024 Epoch  44: total training loss 698.45\n",
            "2019-11-26 17:21:39,025 EPOCH 45\n",
            "2019-11-26 17:21:45,804 Epoch  45 Step:    16500 Batch Loss:     1.638847 Tokens per Sec:     6551, Lr: 0.000300\n",
            "2019-11-26 17:22:17,154 Epoch  45 Step:    16600 Batch Loss:     2.326974 Tokens per Sec:     7207, Lr: 0.000300\n",
            "2019-11-26 17:22:48,131 Epoch  45 Step:    16700 Batch Loss:     1.785748 Tokens per Sec:     7097, Lr: 0.000300\n",
            "2019-11-26 17:23:19,315 Epoch  45 Step:    16800 Batch Loss:     0.960971 Tokens per Sec:     7036, Lr: 0.000300\n",
            "2019-11-26 17:23:36,044 Epoch  45: total training loss 698.68\n",
            "2019-11-26 17:23:36,044 EPOCH 46\n",
            "2019-11-26 17:23:50,802 Epoch  46 Step:    16900 Batch Loss:     1.390077 Tokens per Sec:     7155, Lr: 0.000300\n",
            "2019-11-26 17:24:21,979 Epoch  46 Step:    17000 Batch Loss:     1.922110 Tokens per Sec:     7148, Lr: 0.000300\n",
            "2019-11-26 17:24:52,890 Epoch  46 Step:    17100 Batch Loss:     1.820265 Tokens per Sec:     7002, Lr: 0.000300\n",
            "2019-11-26 17:25:24,097 Epoch  46 Step:    17200 Batch Loss:     1.934819 Tokens per Sec:     7049, Lr: 0.000300\n",
            "2019-11-26 17:25:33,314 Epoch  46: total training loss 695.58\n",
            "2019-11-26 17:25:33,315 EPOCH 47\n",
            "2019-11-26 17:25:55,295 Epoch  47 Step:    17300 Batch Loss:     2.268379 Tokens per Sec:     7047, Lr: 0.000300\n",
            "2019-11-26 17:26:26,338 Epoch  47 Step:    17400 Batch Loss:     1.418422 Tokens per Sec:     7181, Lr: 0.000300\n",
            "2019-11-26 17:26:57,713 Epoch  47 Step:    17500 Batch Loss:     2.127692 Tokens per Sec:     7048, Lr: 0.000300\n",
            "2019-11-26 17:27:29,024 Epoch  47 Step:    17600 Batch Loss:     1.155353 Tokens per Sec:     7177, Lr: 0.000300\n",
            "2019-11-26 17:27:29,801 Epoch  47: total training loss 682.18\n",
            "2019-11-26 17:27:29,801 EPOCH 48\n",
            "2019-11-26 17:28:00,265 Epoch  48 Step:    17700 Batch Loss:     2.210001 Tokens per Sec:     7172, Lr: 0.000300\n",
            "2019-11-26 17:28:31,473 Epoch  48 Step:    17800 Batch Loss:     1.849171 Tokens per Sec:     7120, Lr: 0.000300\n",
            "2019-11-26 17:29:02,788 Epoch  48 Step:    17900 Batch Loss:     1.683955 Tokens per Sec:     7004, Lr: 0.000300\n",
            "2019-11-26 17:29:26,315 Epoch  48: total training loss 676.87\n",
            "2019-11-26 17:29:26,315 EPOCH 49\n",
            "2019-11-26 17:29:33,984 Epoch  49 Step:    18000 Batch Loss:     2.081552 Tokens per Sec:     7268, Lr: 0.000300\n",
            "2019-11-26 17:31:13,696 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 17:31:13,696 Saving new checkpoint.\n",
            "2019-11-26 17:31:14,013 Example #0\n",
            "2019-11-26 17:31:14,014 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 17:31:14,014 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 17:31:14,014 \tHypothesis: Bosenyi kgahlanong le bothata, go bohlokwa gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ba baso ka moka ba bathobaso ka nageng ya rena.\n",
            "2019-11-26 17:31:14,014 Example #1\n",
            "2019-11-26 17:31:14,014 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 17:31:14,014 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 17:31:14,014 \tHypothesis: Kabinete e ile ya feletša Proteve bjalo ka sehlophatšhomo sa dikgopolo tše di sa tšwago go feta mo dibekeng tše di fetilego.\n",
            "2019-11-26 17:31:14,014 Example #2\n",
            "2019-11-26 17:31:14,015 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:31:14,015 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:31:14,015 \tHypothesis: Kgopelo ya go godišwa ga boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 17:31:14,015 Example #3\n",
            "2019-11-26 17:31:14,015 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 17:31:14,015 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 17:31:14,015 \tHypothesis: Molawana wa rena o fa tshedimošo ka ge e ba e ba e ba e gona ge e ba e le gona.\n",
            "2019-11-26 17:31:14,015 Validation result (greedy) at epoch  49, step    18000: bleu:  13.00, loss: 73475.7188, ppl:   9.1613, duration: 100.0312s\n",
            "2019-11-26 17:31:45,113 Epoch  49 Step:    18100 Batch Loss:     1.039702 Tokens per Sec:     6957, Lr: 0.000300\n",
            "2019-11-26 17:32:16,374 Epoch  49 Step:    18200 Batch Loss:     1.947088 Tokens per Sec:     7215, Lr: 0.000300\n",
            "2019-11-26 17:32:47,771 Epoch  49 Step:    18300 Batch Loss:     2.130430 Tokens per Sec:     7114, Lr: 0.000300\n",
            "2019-11-26 17:33:03,193 Epoch  49: total training loss 673.87\n",
            "2019-11-26 17:33:03,193 EPOCH 50\n",
            "2019-11-26 17:33:18,713 Epoch  50 Step:    18400 Batch Loss:     1.979606 Tokens per Sec:     7415, Lr: 0.000300\n",
            "2019-11-26 17:33:49,520 Epoch  50 Step:    18500 Batch Loss:     2.006894 Tokens per Sec:     7108, Lr: 0.000300\n",
            "2019-11-26 17:34:20,416 Epoch  50 Step:    18600 Batch Loss:     1.999812 Tokens per Sec:     6939, Lr: 0.000300\n",
            "2019-11-26 17:34:51,541 Epoch  50 Step:    18700 Batch Loss:     1.809719 Tokens per Sec:     6980, Lr: 0.000300\n",
            "2019-11-26 17:35:00,153 Epoch  50: total training loss 670.08\n",
            "2019-11-26 17:35:00,154 EPOCH 51\n",
            "2019-11-26 17:35:23,101 Epoch  51 Step:    18800 Batch Loss:     2.014594 Tokens per Sec:     7101, Lr: 0.000300\n",
            "2019-11-26 17:35:54,323 Epoch  51 Step:    18900 Batch Loss:     1.640009 Tokens per Sec:     7129, Lr: 0.000300\n",
            "2019-11-26 17:36:25,519 Epoch  51 Step:    19000 Batch Loss:     2.075498 Tokens per Sec:     6936, Lr: 0.000300\n",
            "2019-11-26 17:36:56,703 Epoch  51 Step:    19100 Batch Loss:     2.038496 Tokens per Sec:     7198, Lr: 0.000300\n",
            "2019-11-26 17:36:57,166 Epoch  51: total training loss 663.66\n",
            "2019-11-26 17:36:57,166 EPOCH 52\n",
            "2019-11-26 17:37:27,363 Epoch  52 Step:    19200 Batch Loss:     1.668700 Tokens per Sec:     6948, Lr: 0.000300\n",
            "2019-11-26 17:37:58,766 Epoch  52 Step:    19300 Batch Loss:     0.782651 Tokens per Sec:     7263, Lr: 0.000300\n",
            "2019-11-26 17:38:30,113 Epoch  52 Step:    19400 Batch Loss:     1.734741 Tokens per Sec:     7161, Lr: 0.000300\n",
            "2019-11-26 17:38:53,781 Epoch  52: total training loss 656.09\n",
            "2019-11-26 17:38:53,781 EPOCH 53\n",
            "2019-11-26 17:39:01,269 Epoch  53 Step:    19500 Batch Loss:     2.075530 Tokens per Sec:     7002, Lr: 0.000300\n",
            "2019-11-26 17:39:32,655 Epoch  53 Step:    19600 Batch Loss:     1.673109 Tokens per Sec:     7237, Lr: 0.000300\n",
            "2019-11-26 17:40:03,538 Epoch  53 Step:    19700 Batch Loss:     1.023071 Tokens per Sec:     6927, Lr: 0.000300\n",
            "2019-11-26 17:40:34,479 Epoch  53 Step:    19800 Batch Loss:     1.770991 Tokens per Sec:     7079, Lr: 0.000300\n",
            "2019-11-26 17:40:51,032 Epoch  53: total training loss 656.04\n",
            "2019-11-26 17:40:51,032 EPOCH 54\n",
            "2019-11-26 17:41:05,564 Epoch  54 Step:    19900 Batch Loss:     1.088404 Tokens per Sec:     7023, Lr: 0.000300\n",
            "2019-11-26 17:41:36,831 Epoch  54 Step:    20000 Batch Loss:     1.839398 Tokens per Sec:     7039, Lr: 0.000300\n",
            "2019-11-26 17:43:16,523 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 17:43:16,523 Saving new checkpoint.\n",
            "2019-11-26 17:43:16,809 Example #0\n",
            "2019-11-26 17:43:16,809 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 17:43:16,809 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 17:43:16,810 \tHypothesis: Bosenyi kgahlanong le bothata, go bohlokwa gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ka moka ka nageng ya rena.\n",
            "2019-11-26 17:43:16,810 Example #1\n",
            "2019-11-26 17:43:16,810 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 17:43:16,810 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 17:43:16,810 \tHypothesis: Kabinete le yona e ile ya lebogiša Protšeke ya go dira gore dikgopolo tša go se be le tše ntši tša Seisimane tše di fetilego .\n",
            "2019-11-26 17:43:16,810 Example #2\n",
            "2019-11-26 17:43:16,811 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:43:16,811 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:43:16,811 \tHypothesis: Kgopelo ya go godišwa ga boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 17:43:16,811 Example #3\n",
            "2019-11-26 17:43:16,811 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 17:43:16,811 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 17:43:16,812 \tHypothesis: Molaotshepetšo wa rena o fa tshedimošo ka ge e ba gona e ba gona.\n",
            "2019-11-26 17:43:16,812 Validation result (greedy) at epoch  54, step    20000: bleu:  14.78, loss: 72637.5469, ppl:   8.9328, duration: 99.9800s\n",
            "2019-11-26 17:43:48,183 Epoch  54 Step:    20100 Batch Loss:     1.663085 Tokens per Sec:     7239, Lr: 0.000300\n",
            "2019-11-26 17:44:19,354 Epoch  54 Step:    20200 Batch Loss:     1.887033 Tokens per Sec:     7071, Lr: 0.000300\n",
            "2019-11-26 17:44:27,475 Epoch  54: total training loss 642.56\n",
            "2019-11-26 17:44:27,476 EPOCH 55\n",
            "2019-11-26 17:44:50,726 Epoch  55 Step:    20300 Batch Loss:     1.481136 Tokens per Sec:     7283, Lr: 0.000300\n",
            "2019-11-26 17:45:21,819 Epoch  55 Step:    20400 Batch Loss:     1.827910 Tokens per Sec:     6988, Lr: 0.000300\n",
            "2019-11-26 17:45:53,062 Epoch  55 Step:    20500 Batch Loss:     1.713349 Tokens per Sec:     7127, Lr: 0.000300\n",
            "2019-11-26 17:46:24,000 Epoch  55 Step:    20600 Batch Loss:     1.991260 Tokens per Sec:     7099, Lr: 0.000300\n",
            "2019-11-26 17:46:24,002 Epoch  55: total training loss 638.92\n",
            "2019-11-26 17:46:24,002 EPOCH 56\n",
            "2019-11-26 17:46:55,447 Epoch  56 Step:    20700 Batch Loss:     2.057329 Tokens per Sec:     7161, Lr: 0.000300\n",
            "2019-11-26 17:47:26,537 Epoch  56 Step:    20800 Batch Loss:     1.047325 Tokens per Sec:     7156, Lr: 0.000300\n",
            "2019-11-26 17:47:57,735 Epoch  56 Step:    20900 Batch Loss:     2.054070 Tokens per Sec:     7104, Lr: 0.000300\n",
            "2019-11-26 17:48:20,707 Epoch  56: total training loss 633.59\n",
            "2019-11-26 17:48:20,708 EPOCH 57\n",
            "2019-11-26 17:48:28,990 Epoch  57 Step:    21000 Batch Loss:     1.789886 Tokens per Sec:     7037, Lr: 0.000300\n",
            "2019-11-26 17:49:00,145 Epoch  57 Step:    21100 Batch Loss:     1.761622 Tokens per Sec:     7158, Lr: 0.000300\n",
            "2019-11-26 17:49:31,288 Epoch  57 Step:    21200 Batch Loss:     1.686143 Tokens per Sec:     6886, Lr: 0.000300\n",
            "2019-11-26 17:50:02,776 Epoch  57 Step:    21300 Batch Loss:     1.835996 Tokens per Sec:     7291, Lr: 0.000300\n",
            "2019-11-26 17:50:17,554 Epoch  57: total training loss 627.17\n",
            "2019-11-26 17:50:17,554 EPOCH 58\n",
            "2019-11-26 17:50:34,100 Epoch  58 Step:    21400 Batch Loss:     1.501171 Tokens per Sec:     7178, Lr: 0.000300\n",
            "2019-11-26 17:51:05,144 Epoch  58 Step:    21500 Batch Loss:     0.803616 Tokens per Sec:     7121, Lr: 0.000300\n",
            "2019-11-26 17:51:36,322 Epoch  58 Step:    21600 Batch Loss:     1.602395 Tokens per Sec:     7079, Lr: 0.000300\n",
            "2019-11-26 17:52:07,753 Epoch  58 Step:    21700 Batch Loss:     2.105183 Tokens per Sec:     7100, Lr: 0.000300\n",
            "2019-11-26 17:52:14,205 Epoch  58: total training loss 624.61\n",
            "2019-11-26 17:52:14,205 EPOCH 59\n",
            "2019-11-26 17:52:39,038 Epoch  59 Step:    21800 Batch Loss:     1.840311 Tokens per Sec:     7096, Lr: 0.000300\n",
            "2019-11-26 17:53:10,205 Epoch  59 Step:    21900 Batch Loss:     0.807408 Tokens per Sec:     7019, Lr: 0.000300\n",
            "2019-11-26 17:53:41,654 Epoch  59 Step:    22000 Batch Loss:     1.146837 Tokens per Sec:     7090, Lr: 0.000300\n",
            "2019-11-26 17:55:21,445 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 17:55:21,446 Saving new checkpoint.\n",
            "2019-11-26 17:55:21,733 Example #0\n",
            "2019-11-26 17:55:21,734 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 17:55:21,734 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 17:55:21,734 \tHypothesis: Bosenyi kgahlanong le bothata, go a kgonagala gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ka moka ba baso ka nageng ya rena.\n",
            "2019-11-26 17:55:21,734 Example #1\n",
            "2019-11-26 17:55:21,735 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 17:55:21,735 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 17:55:21,735 \tHypothesis: Kabinete le yona e ile ya lebogiša Proteas ya go dira gore dikgopolo tše di ka bago gona ka Seisimane kgahlanong le dinaga tše lesomehlano tša go feta.\n",
            "2019-11-26 17:55:21,735 Example #2\n",
            "2019-11-26 17:55:21,735 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:55:21,735 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 17:55:21,736 \tHypothesis: Kgopelo ya go swarwa ga boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 17:55:21,736 Example #3\n",
            "2019-11-26 17:55:21,736 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 17:55:21,736 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 17:55:21,736 \tHypothesis: Molaotshepetšo wa rena ke go neelana ka tshedimošo ge e ba gona e ba gona.\n",
            "2019-11-26 17:55:21,736 Validation result (greedy) at epoch  59, step    22000: bleu:  15.04, loss: 71780.0391, ppl:   8.7048, duration: 100.0824s\n",
            "2019-11-26 17:55:51,129 Epoch  59: total training loss 617.68\n",
            "2019-11-26 17:55:51,129 EPOCH 60\n",
            "2019-11-26 17:55:53,035 Epoch  60 Step:    22100 Batch Loss:     1.547916 Tokens per Sec:     6581, Lr: 0.000300\n",
            "2019-11-26 17:56:24,254 Epoch  60 Step:    22200 Batch Loss:     1.328555 Tokens per Sec:     7174, Lr: 0.000300\n",
            "2019-11-26 17:56:55,316 Epoch  60 Step:    22300 Batch Loss:     1.860139 Tokens per Sec:     7068, Lr: 0.000300\n",
            "2019-11-26 17:57:26,442 Epoch  60 Step:    22400 Batch Loss:     1.763628 Tokens per Sec:     6952, Lr: 0.000300\n",
            "2019-11-26 17:57:47,941 Epoch  60: total training loss 616.88\n",
            "2019-11-26 17:57:47,942 EPOCH 61\n",
            "2019-11-26 17:57:57,549 Epoch  61 Step:    22500 Batch Loss:     1.923210 Tokens per Sec:     6736, Lr: 0.000300\n",
            "2019-11-26 17:58:28,965 Epoch  61 Step:    22600 Batch Loss:     1.793148 Tokens per Sec:     7175, Lr: 0.000300\n",
            "2019-11-26 17:58:59,851 Epoch  61 Step:    22700 Batch Loss:     1.601910 Tokens per Sec:     6980, Lr: 0.000300\n",
            "2019-11-26 17:59:30,958 Epoch  61 Step:    22800 Batch Loss:     1.838136 Tokens per Sec:     7171, Lr: 0.000300\n",
            "2019-11-26 17:59:45,039 Epoch  61: total training loss 614.52\n",
            "2019-11-26 17:59:45,040 EPOCH 62\n",
            "2019-11-26 18:00:02,185 Epoch  62 Step:    22900 Batch Loss:     1.580576 Tokens per Sec:     7224, Lr: 0.000300\n",
            "2019-11-26 18:00:33,407 Epoch  62 Step:    23000 Batch Loss:     1.584725 Tokens per Sec:     7054, Lr: 0.000300\n",
            "2019-11-26 18:01:04,600 Epoch  62 Step:    23100 Batch Loss:     1.283049 Tokens per Sec:     7091, Lr: 0.000300\n",
            "2019-11-26 18:01:35,813 Epoch  62 Step:    23200 Batch Loss:     1.355496 Tokens per Sec:     7204, Lr: 0.000300\n",
            "2019-11-26 18:01:41,600 Epoch  62: total training loss 607.18\n",
            "2019-11-26 18:01:41,600 EPOCH 63\n",
            "2019-11-26 18:02:07,122 Epoch  63 Step:    23300 Batch Loss:     1.654276 Tokens per Sec:     7210, Lr: 0.000300\n",
            "2019-11-26 18:02:38,346 Epoch  63 Step:    23400 Batch Loss:     1.913656 Tokens per Sec:     7052, Lr: 0.000300\n",
            "2019-11-26 18:03:09,562 Epoch  63 Step:    23500 Batch Loss:     1.551469 Tokens per Sec:     7043, Lr: 0.000300\n",
            "2019-11-26 18:03:38,082 Epoch  63: total training loss 600.65\n",
            "2019-11-26 18:03:38,082 EPOCH 64\n",
            "2019-11-26 18:03:40,965 Epoch  64 Step:    23600 Batch Loss:     1.871647 Tokens per Sec:     7291, Lr: 0.000300\n",
            "2019-11-26 18:04:12,003 Epoch  64 Step:    23700 Batch Loss:     0.812826 Tokens per Sec:     6997, Lr: 0.000300\n",
            "2019-11-26 18:04:43,108 Epoch  64 Step:    23800 Batch Loss:     2.039389 Tokens per Sec:     6977, Lr: 0.000300\n",
            "2019-11-26 18:05:14,390 Epoch  64 Step:    23900 Batch Loss:     1.539271 Tokens per Sec:     7146, Lr: 0.000300\n",
            "2019-11-26 18:05:35,217 Epoch  64: total training loss 600.68\n",
            "2019-11-26 18:05:35,217 EPOCH 65\n",
            "2019-11-26 18:05:46,134 Epoch  65 Step:    24000 Batch Loss:     1.367700 Tokens per Sec:     7272, Lr: 0.000300\n",
            "2019-11-26 18:07:25,947 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 18:07:25,947 Saving new checkpoint.\n",
            "2019-11-26 18:07:26,243 Example #0\n",
            "2019-11-26 18:07:26,246 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 18:07:26,246 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 18:07:26,248 \tHypothesis: Bothata kgahlanong le bothata, go bohlokwa gore tshepedišo ya kgethologanyo e tla ba le seabe sa batho ka moka ka nageng ya rena.\n",
            "2019-11-26 18:07:26,248 Example #1\n",
            "2019-11-26 18:07:26,249 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 18:07:26,249 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 18:07:26,249 \tHypothesis: Kabinete e lebogile Proteas ye e lego gona ya dikgopolo tša go dira gore go be le ditoropokgolo tše di lego kgauswi le Seipe.\n",
            "2019-11-26 18:07:26,249 Example #2\n",
            "2019-11-26 18:07:26,250 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:07:26,250 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:07:26,250 \tHypothesis: Kgopelo ya go hwetša gape maemo a boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 18:07:26,250 Example #3\n",
            "2019-11-26 18:07:26,251 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 18:07:26,251 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 18:07:26,251 \tHypothesis: Molawana wa rena ke go fa tshedimošo ge e ba e ba e ba gona le gona ge e le gore e ba gona.\n",
            "2019-11-26 18:07:26,251 Validation result (greedy) at epoch  65, step    24000: bleu:  16.54, loss: 71055.6797, ppl:   8.5168, duration: 100.1162s\n",
            "2019-11-26 18:07:57,087 Epoch  65 Step:    24100 Batch Loss:     1.669462 Tokens per Sec:     6970, Lr: 0.000300\n",
            "2019-11-26 18:08:28,117 Epoch  65 Step:    24200 Batch Loss:     1.499993 Tokens per Sec:     7140, Lr: 0.000300\n",
            "2019-11-26 18:08:59,303 Epoch  65 Step:    24300 Batch Loss:     1.936673 Tokens per Sec:     7138, Lr: 0.000300\n",
            "2019-11-26 18:09:12,106 Epoch  65: total training loss 595.30\n",
            "2019-11-26 18:09:12,106 EPOCH 66\n",
            "2019-11-26 18:09:30,714 Epoch  66 Step:    24400 Batch Loss:     1.937520 Tokens per Sec:     7246, Lr: 0.000300\n",
            "2019-11-26 18:10:02,066 Epoch  66 Step:    24500 Batch Loss:     1.711078 Tokens per Sec:     7138, Lr: 0.000300\n",
            "2019-11-26 18:10:33,198 Epoch  66 Step:    24600 Batch Loss:     1.478998 Tokens per Sec:     7057, Lr: 0.000300\n",
            "2019-11-26 18:11:04,277 Epoch  66 Step:    24700 Batch Loss:     1.754385 Tokens per Sec:     7029, Lr: 0.000300\n",
            "2019-11-26 18:11:08,854 Epoch  66: total training loss 592.25\n",
            "2019-11-26 18:11:08,854 EPOCH 67\n",
            "2019-11-26 18:11:35,370 Epoch  67 Step:    24800 Batch Loss:     1.134527 Tokens per Sec:     7253, Lr: 0.000300\n",
            "2019-11-26 18:12:06,381 Epoch  67 Step:    24900 Batch Loss:     1.043904 Tokens per Sec:     7085, Lr: 0.000300\n",
            "2019-11-26 18:12:37,908 Epoch  67 Step:    25000 Batch Loss:     2.013454 Tokens per Sec:     7167, Lr: 0.000300\n",
            "2019-11-26 18:13:04,950 Epoch  67: total training loss 583.99\n",
            "2019-11-26 18:13:04,950 EPOCH 68\n",
            "2019-11-26 18:13:08,981 Epoch  68 Step:    25100 Batch Loss:     1.521513 Tokens per Sec:     6760, Lr: 0.000300\n",
            "2019-11-26 18:13:40,440 Epoch  68 Step:    25200 Batch Loss:     1.697865 Tokens per Sec:     7202, Lr: 0.000300\n",
            "2019-11-26 18:14:11,448 Epoch  68 Step:    25300 Batch Loss:     0.741487 Tokens per Sec:     7040, Lr: 0.000300\n",
            "2019-11-26 18:14:42,633 Epoch  68 Step:    25400 Batch Loss:     1.840793 Tokens per Sec:     7155, Lr: 0.000300\n",
            "2019-11-26 18:15:02,092 Epoch  68: total training loss 584.75\n",
            "2019-11-26 18:15:02,092 EPOCH 69\n",
            "2019-11-26 18:15:13,621 Epoch  69 Step:    25500 Batch Loss:     1.504202 Tokens per Sec:     6896, Lr: 0.000300\n",
            "2019-11-26 18:15:44,388 Epoch  69 Step:    25600 Batch Loss:     1.488906 Tokens per Sec:     7089, Lr: 0.000300\n",
            "2019-11-26 18:16:15,485 Epoch  69 Step:    25700 Batch Loss:     1.944386 Tokens per Sec:     6992, Lr: 0.000300\n",
            "2019-11-26 18:16:46,902 Epoch  69 Step:    25800 Batch Loss:     1.346239 Tokens per Sec:     7194, Lr: 0.000300\n",
            "2019-11-26 18:16:59,107 Epoch  69: total training loss 583.62\n",
            "2019-11-26 18:16:59,107 EPOCH 70\n",
            "2019-11-26 18:17:18,052 Epoch  70 Step:    25900 Batch Loss:     1.778444 Tokens per Sec:     7051, Lr: 0.000300\n",
            "2019-11-26 18:17:49,641 Epoch  70 Step:    26000 Batch Loss:     1.066573 Tokens per Sec:     7277, Lr: 0.000300\n",
            "2019-11-26 18:19:29,305 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 18:19:29,305 Saving new checkpoint.\n",
            "2019-11-26 18:19:29,816 Example #0\n",
            "2019-11-26 18:19:29,816 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 18:19:29,816 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 18:19:29,817 \tHypothesis: Bosenyi kgahlanong le bothata, go bohlokwa gore tshepedišo ya kgethologanyo e tla ba le seabe sa go batho ba bathobaso ka nageng ya rena.\n",
            "2019-11-26 18:19:29,817 Example #1\n",
            "2019-11-26 18:19:29,817 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 18:19:29,817 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 18:19:29,817 \tHypothesis: Kabinete gape e lebogišitše Proteas ka ge e le gore go na le dikgopolo tše mmalwa tša go feta mo dibekeng tše di fetilego.\n",
            "2019-11-26 18:19:29,817 Example #2\n",
            "2019-11-26 18:19:29,818 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:19:29,818 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:19:29,818 \tHypothesis: Kgopelo ya go hwetša gape maemo a boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 18:19:29,819 Example #3\n",
            "2019-11-26 18:19:29,819 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 18:19:29,819 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 18:19:29,819 \tHypothesis: Molaotshepetšo wa rena ke go neelana ka tshedimošo le ge e le gore e ba gona.\n",
            "2019-11-26 18:19:29,819 Validation result (greedy) at epoch  70, step    26000: bleu:  16.94, loss: 70854.2344, ppl:   8.4652, duration: 100.1777s\n",
            "2019-11-26 18:20:01,015 Epoch  70 Step:    26100 Batch Loss:     1.895302 Tokens per Sec:     7077, Lr: 0.000300\n",
            "2019-11-26 18:20:32,160 Epoch  70 Step:    26200 Batch Loss:     1.925363 Tokens per Sec:     7107, Lr: 0.000300\n",
            "2019-11-26 18:20:35,709 Epoch  70: total training loss 573.08\n",
            "2019-11-26 18:20:35,710 EPOCH 71\n",
            "2019-11-26 18:21:03,073 Epoch  71 Step:    26300 Batch Loss:     1.063808 Tokens per Sec:     7039, Lr: 0.000300\n",
            "2019-11-26 18:21:34,642 Epoch  71 Step:    26400 Batch Loss:     1.840748 Tokens per Sec:     7144, Lr: 0.000300\n",
            "2019-11-26 18:22:05,646 Epoch  71 Step:    26500 Batch Loss:     1.512532 Tokens per Sec:     7036, Lr: 0.000300\n",
            "2019-11-26 18:22:32,694 Epoch  71: total training loss 571.22\n",
            "2019-11-26 18:22:32,695 EPOCH 72\n",
            "2019-11-26 18:22:37,078 Epoch  72 Step:    26600 Batch Loss:     1.468149 Tokens per Sec:     6915, Lr: 0.000300\n",
            "2019-11-26 18:23:08,573 Epoch  72 Step:    26700 Batch Loss:     1.007459 Tokens per Sec:     7195, Lr: 0.000300\n",
            "2019-11-26 18:23:39,259 Epoch  72 Step:    26800 Batch Loss:     1.393209 Tokens per Sec:     6906, Lr: 0.000300\n",
            "2019-11-26 18:24:10,596 Epoch  72 Step:    26900 Batch Loss:     1.544017 Tokens per Sec:     7114, Lr: 0.000300\n",
            "2019-11-26 18:24:30,155 Epoch  72: total training loss 570.77\n",
            "2019-11-26 18:24:30,155 EPOCH 73\n",
            "2019-11-26 18:24:41,424 Epoch  73 Step:    27000 Batch Loss:     1.580617 Tokens per Sec:     6804, Lr: 0.000300\n",
            "2019-11-26 18:25:12,991 Epoch  73 Step:    27100 Batch Loss:     1.350520 Tokens per Sec:     7212, Lr: 0.000300\n",
            "2019-11-26 18:25:44,294 Epoch  73 Step:    27200 Batch Loss:     1.542511 Tokens per Sec:     7115, Lr: 0.000300\n",
            "2019-11-26 18:26:15,606 Epoch  73 Step:    27300 Batch Loss:     1.894775 Tokens per Sec:     7231, Lr: 0.000300\n",
            "2019-11-26 18:26:26,443 Epoch  73: total training loss 561.79\n",
            "2019-11-26 18:26:26,443 EPOCH 74\n",
            "2019-11-26 18:26:46,651 Epoch  74 Step:    27400 Batch Loss:     1.767156 Tokens per Sec:     7105, Lr: 0.000300\n",
            "2019-11-26 18:27:17,912 Epoch  74 Step:    27500 Batch Loss:     1.788654 Tokens per Sec:     7114, Lr: 0.000300\n",
            "2019-11-26 18:27:49,379 Epoch  74 Step:    27600 Batch Loss:     1.447718 Tokens per Sec:     7293, Lr: 0.000300\n",
            "2019-11-26 18:28:20,336 Epoch  74 Step:    27700 Batch Loss:     1.743050 Tokens per Sec:     6969, Lr: 0.000300\n",
            "2019-11-26 18:28:23,043 Epoch  74: total training loss 560.52\n",
            "2019-11-26 18:28:23,044 EPOCH 75\n",
            "2019-11-26 18:28:51,321 Epoch  75 Step:    27800 Batch Loss:     1.779340 Tokens per Sec:     6992, Lr: 0.000300\n",
            "2019-11-26 18:29:22,192 Epoch  75 Step:    27900 Batch Loss:     1.126480 Tokens per Sec:     7055, Lr: 0.000300\n",
            "2019-11-26 18:29:53,585 Epoch  75 Step:    28000 Batch Loss:     1.512679 Tokens per Sec:     7167, Lr: 0.000300\n",
            "2019-11-26 18:31:33,270 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 18:31:33,270 Saving new checkpoint.\n",
            "2019-11-26 18:31:33,609 Example #0\n",
            "2019-11-26 18:31:33,610 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 18:31:33,610 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 18:31:33,610 \tHypothesis: Bothata kgahlanong le bothata, go a kgontšha gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ka moka ba bathobaso ka nageng.\n",
            "2019-11-26 18:31:33,610 Example #1\n",
            "2019-11-26 18:31:33,611 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 18:31:33,612 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 18:31:33,612 \tHypothesis: Kabinete gape e lebogišitše Proteas ka ge e le gore e be e le gore e be e le gore e be e le gore e be e le gore e be e le gore e be e le go na le bonnete bja gore e be e le go se na bonnete bja go se gwa ba se sengwe.\n",
            "2019-11-26 18:31:33,612 Example #2\n",
            "2019-11-26 18:31:33,612 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:31:33,613 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:31:33,613 \tHypothesis: Kgopelo ya go tloša boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 18:31:33,613 Example #3\n",
            "2019-11-26 18:31:33,613 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 18:31:33,614 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 18:31:33,614 \tHypothesis: Molaotshepetšo wa rena ke go fa tshedimošo ge e ba gona ge e ba gona.\n",
            "2019-11-26 18:31:33,615 Validation result (greedy) at epoch  75, step    28000: bleu:  18.12, loss: 69797.8203, ppl:   8.1999, duration: 100.0290s\n",
            "2019-11-26 18:32:00,570 Epoch  75: total training loss 564.19\n",
            "2019-11-26 18:32:00,570 EPOCH 76\n",
            "2019-11-26 18:32:04,821 Epoch  76 Step:    28100 Batch Loss:     1.632364 Tokens per Sec:     7457, Lr: 0.000300\n",
            "2019-11-26 18:32:36,020 Epoch  76 Step:    28200 Batch Loss:     1.827327 Tokens per Sec:     7130, Lr: 0.000300\n",
            "2019-11-26 18:33:07,194 Epoch  76 Step:    28300 Batch Loss:     0.615108 Tokens per Sec:     7041, Lr: 0.000300\n",
            "2019-11-26 18:33:38,869 Epoch  76 Step:    28400 Batch Loss:     1.869656 Tokens per Sec:     7243, Lr: 0.000300\n",
            "2019-11-26 18:33:57,061 Epoch  76: total training loss 552.00\n",
            "2019-11-26 18:33:57,062 EPOCH 77\n",
            "2019-11-26 18:34:09,768 Epoch  77 Step:    28500 Batch Loss:     1.136177 Tokens per Sec:     7061, Lr: 0.000300\n",
            "2019-11-26 18:34:41,202 Epoch  77 Step:    28600 Batch Loss:     1.657164 Tokens per Sec:     7072, Lr: 0.000300\n",
            "2019-11-26 18:35:12,455 Epoch  77 Step:    28700 Batch Loss:     1.659379 Tokens per Sec:     7054, Lr: 0.000300\n",
            "2019-11-26 18:35:43,511 Epoch  77 Step:    28800 Batch Loss:     1.484225 Tokens per Sec:     7178, Lr: 0.000300\n",
            "2019-11-26 18:35:53,792 Epoch  77: total training loss 550.76\n",
            "2019-11-26 18:35:53,792 EPOCH 78\n",
            "2019-11-26 18:36:14,766 Epoch  78 Step:    28900 Batch Loss:     1.903091 Tokens per Sec:     7235, Lr: 0.000300\n",
            "2019-11-26 18:36:45,735 Epoch  78 Step:    29000 Batch Loss:     1.435938 Tokens per Sec:     6941, Lr: 0.000300\n",
            "2019-11-26 18:37:16,768 Epoch  78 Step:    29100 Batch Loss:     1.100330 Tokens per Sec:     7070, Lr: 0.000300\n",
            "2019-11-26 18:37:48,546 Epoch  78 Step:    29200 Batch Loss:     1.382899 Tokens per Sec:     7200, Lr: 0.000300\n",
            "2019-11-26 18:37:50,648 Epoch  78: total training loss 548.46\n",
            "2019-11-26 18:37:50,648 EPOCH 79\n",
            "2019-11-26 18:38:19,554 Epoch  79 Step:    29300 Batch Loss:     1.656077 Tokens per Sec:     7027, Lr: 0.000300\n",
            "2019-11-26 18:38:50,793 Epoch  79 Step:    29400 Batch Loss:     1.480344 Tokens per Sec:     7049, Lr: 0.000300\n",
            "2019-11-26 18:39:21,980 Epoch  79 Step:    29500 Batch Loss:     1.451218 Tokens per Sec:     7101, Lr: 0.000300\n",
            "2019-11-26 18:39:47,824 Epoch  79: total training loss 545.55\n",
            "2019-11-26 18:39:47,824 EPOCH 80\n",
            "2019-11-26 18:39:53,687 Epoch  80 Step:    29600 Batch Loss:     1.885525 Tokens per Sec:     7553, Lr: 0.000300\n",
            "2019-11-26 18:40:24,875 Epoch  80 Step:    29700 Batch Loss:     1.411130 Tokens per Sec:     6992, Lr: 0.000300\n",
            "2019-11-26 18:40:56,013 Epoch  80 Step:    29800 Batch Loss:     1.618700 Tokens per Sec:     6989, Lr: 0.000300\n",
            "2019-11-26 18:41:27,580 Epoch  80 Step:    29900 Batch Loss:     1.558182 Tokens per Sec:     7274, Lr: 0.000300\n",
            "2019-11-26 18:41:44,859 Epoch  80: total training loss 541.39\n",
            "2019-11-26 18:41:44,859 EPOCH 81\n",
            "2019-11-26 18:41:58,536 Epoch  81 Step:    30000 Batch Loss:     1.336547 Tokens per Sec:     7237, Lr: 0.000300\n",
            "2019-11-26 18:43:38,206 Example #0\n",
            "2019-11-26 18:43:38,206 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 18:43:38,206 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 18:43:38,206 \tHypothesis: Bosenyi kgahlanong le bothata, go a kgonagala gore tshepedišo ya kgethologanyo e tla ba le seabe ka botlalo ka go batho ba bathobaso ka nageng.\n",
            "2019-11-26 18:43:38,207 Example #1\n",
            "2019-11-26 18:43:38,207 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 18:43:38,207 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 18:43:38,207 \tHypothesis: Kabinete le yona e lebogile Proteas ka ge e le gore dikgopolo tše di sa kgahlišego tša go swana le England mo bekeng ye e fetilego.\n",
            "2019-11-26 18:43:38,207 Example #2\n",
            "2019-11-26 18:43:38,207 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:43:38,207 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:43:38,207 \tHypothesis: Kgopelo ya go hwetša gape boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 18:43:38,208 Example #3\n",
            "2019-11-26 18:43:38,208 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 18:43:38,208 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 18:43:38,208 \tHypothesis: Pholisi ya rena ke go fa tshedimošo ge e le gore e ba gona.\n",
            "2019-11-26 18:43:38,208 Validation result (greedy) at epoch  81, step    30000: bleu:  18.49, loss: 70076.9766, ppl:   8.2692, duration: 99.6719s\n",
            "2019-11-26 18:44:09,538 Epoch  81 Step:    30100 Batch Loss:     1.236897 Tokens per Sec:     7057, Lr: 0.000300\n",
            "2019-11-26 18:44:40,848 Epoch  81 Step:    30200 Batch Loss:     1.098529 Tokens per Sec:     7158, Lr: 0.000300\n",
            "2019-11-26 18:45:12,075 Epoch  81 Step:    30300 Batch Loss:     1.612874 Tokens per Sec:     7077, Lr: 0.000300\n",
            "2019-11-26 18:45:21,177 Epoch  81: total training loss 537.92\n",
            "2019-11-26 18:45:21,178 EPOCH 82\n",
            "2019-11-26 18:45:43,556 Epoch  82 Step:    30400 Batch Loss:     1.748863 Tokens per Sec:     7007, Lr: 0.000300\n",
            "2019-11-26 18:46:14,334 Epoch  82 Step:    30500 Batch Loss:     1.393545 Tokens per Sec:     7161, Lr: 0.000300\n",
            "2019-11-26 18:46:45,656 Epoch  82 Step:    30600 Batch Loss:     1.533866 Tokens per Sec:     7108, Lr: 0.000300\n",
            "2019-11-26 18:47:16,910 Epoch  82 Step:    30700 Batch Loss:     1.333227 Tokens per Sec:     7156, Lr: 0.000300\n",
            "2019-11-26 18:47:17,695 Epoch  82: total training loss 535.54\n",
            "2019-11-26 18:47:17,696 EPOCH 83\n",
            "2019-11-26 18:47:47,775 Epoch  83 Step:    30800 Batch Loss:     1.592650 Tokens per Sec:     7009, Lr: 0.000300\n",
            "2019-11-26 18:48:19,235 Epoch  83 Step:    30900 Batch Loss:     1.712885 Tokens per Sec:     7260, Lr: 0.000300\n",
            "2019-11-26 18:48:50,545 Epoch  83 Step:    31000 Batch Loss:     1.792123 Tokens per Sec:     7105, Lr: 0.000300\n",
            "2019-11-26 18:49:14,580 Epoch  83: total training loss 534.72\n",
            "2019-11-26 18:49:14,580 EPOCH 84\n",
            "2019-11-26 18:49:21,329 Epoch  84 Step:    31100 Batch Loss:     1.250563 Tokens per Sec:     6743, Lr: 0.000300\n",
            "2019-11-26 18:49:52,769 Epoch  84 Step:    31200 Batch Loss:     1.589976 Tokens per Sec:     7160, Lr: 0.000300\n",
            "2019-11-26 18:50:23,840 Epoch  84 Step:    31300 Batch Loss:     0.603262 Tokens per Sec:     6999, Lr: 0.000300\n",
            "2019-11-26 18:50:54,950 Epoch  84 Step:    31400 Batch Loss:     1.482901 Tokens per Sec:     7129, Lr: 0.000300\n",
            "2019-11-26 18:51:11,356 Epoch  84: total training loss 529.35\n",
            "2019-11-26 18:51:11,356 EPOCH 85\n",
            "2019-11-26 18:51:26,512 Epoch  85 Step:    31500 Batch Loss:     1.527025 Tokens per Sec:     7169, Lr: 0.000300\n",
            "2019-11-26 18:51:57,843 Epoch  85 Step:    31600 Batch Loss:     1.406240 Tokens per Sec:     7144, Lr: 0.000300\n",
            "2019-11-26 18:52:29,038 Epoch  85 Step:    31700 Batch Loss:     1.651632 Tokens per Sec:     7064, Lr: 0.000300\n",
            "2019-11-26 18:53:00,374 Epoch  85 Step:    31800 Batch Loss:     1.240734 Tokens per Sec:     7192, Lr: 0.000300\n",
            "2019-11-26 18:53:08,135 Epoch  85: total training loss 529.00\n",
            "2019-11-26 18:53:08,135 EPOCH 86\n",
            "2019-11-26 18:53:31,395 Epoch  86 Step:    31900 Batch Loss:     1.721380 Tokens per Sec:     7341, Lr: 0.000300\n",
            "2019-11-26 18:54:02,344 Epoch  86 Step:    32000 Batch Loss:     1.051812 Tokens per Sec:     6989, Lr: 0.000300\n",
            "2019-11-26 18:55:42,007 Example #0\n",
            "2019-11-26 18:55:42,007 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 18:55:42,007 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 18:55:42,007 \tHypothesis: Bosenyi kgahlanong le bothata, go a kgonagala gore tshepedišo ya kgethologanyo e tla ba le seabe sa batho ka moka ka nageng.\n",
            "2019-11-26 18:55:42,007 Example #1\n",
            "2019-11-26 18:55:42,008 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 18:55:42,008 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 18:55:42,008 \tHypothesis: Kabinete le yona e lebogišitše Proteas ka ditsebe tša ditsebe tša go ba le dikgopolo tše di fapafapanego tša go feta tše pedi.\n",
            "2019-11-26 18:55:42,008 Example #2\n",
            "2019-11-26 18:55:42,008 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:55:42,008 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 18:55:42,008 \tHypothesis: Kgopelo ya go hwetša gape maemo a boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 18:55:42,008 Example #3\n",
            "2019-11-26 18:55:42,008 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 18:55:42,008 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 18:55:42,008 \tHypothesis: Pholisi ya rena ke go fana ka tshedimošo ge e ba gona e le gona.\n",
            "2019-11-26 18:55:42,009 Validation result (greedy) at epoch  86, step    32000: bleu:  18.72, loss: 69808.2500, ppl:   8.2024, duration: 99.6645s\n",
            "2019-11-26 18:56:13,084 Epoch  86 Step:    32100 Batch Loss:     1.170709 Tokens per Sec:     7066, Lr: 0.000300\n",
            "2019-11-26 18:56:44,333 Epoch  86 Step:    32200 Batch Loss:     1.274538 Tokens per Sec:     7113, Lr: 0.000300\n",
            "2019-11-26 18:56:44,334 Epoch  86: total training loss 527.30\n",
            "2019-11-26 18:56:44,335 EPOCH 87\n",
            "2019-11-26 18:57:15,720 Epoch  87 Step:    32300 Batch Loss:     1.328041 Tokens per Sec:     7180, Lr: 0.000300\n",
            "2019-11-26 18:57:46,745 Epoch  87 Step:    32400 Batch Loss:     1.158772 Tokens per Sec:     7029, Lr: 0.000300\n",
            "2019-11-26 18:58:17,883 Epoch  87 Step:    32500 Batch Loss:     1.479058 Tokens per Sec:     7256, Lr: 0.000300\n",
            "2019-11-26 18:58:41,011 Epoch  87: total training loss 523.05\n",
            "2019-11-26 18:58:41,011 EPOCH 88\n",
            "2019-11-26 18:58:48,756 Epoch  88 Step:    32600 Batch Loss:     1.506139 Tokens per Sec:     6876, Lr: 0.000300\n",
            "2019-11-26 18:59:20,130 Epoch  88 Step:    32700 Batch Loss:     1.489486 Tokens per Sec:     7173, Lr: 0.000300\n",
            "2019-11-26 18:59:51,449 Epoch  88 Step:    32800 Batch Loss:     1.320836 Tokens per Sec:     7166, Lr: 0.000300\n",
            "2019-11-26 19:00:22,595 Epoch  88 Step:    32900 Batch Loss:     1.567721 Tokens per Sec:     7087, Lr: 0.000300\n",
            "2019-11-26 19:00:37,904 Epoch  88: total training loss 519.63\n",
            "2019-11-26 19:00:37,905 EPOCH 89\n",
            "2019-11-26 19:00:53,831 Epoch  89 Step:    33000 Batch Loss:     1.629923 Tokens per Sec:     7210, Lr: 0.000300\n",
            "2019-11-26 19:01:24,693 Epoch  89 Step:    33100 Batch Loss:     1.504692 Tokens per Sec:     6993, Lr: 0.000300\n",
            "2019-11-26 19:01:55,872 Epoch  89 Step:    33200 Batch Loss:     1.407284 Tokens per Sec:     7119, Lr: 0.000300\n",
            "2019-11-26 19:02:27,070 Epoch  89 Step:    33300 Batch Loss:     1.544786 Tokens per Sec:     7066, Lr: 0.000300\n",
            "2019-11-26 19:02:34,850 Epoch  89: total training loss 518.85\n",
            "2019-11-26 19:02:34,850 EPOCH 90\n",
            "2019-11-26 19:02:58,596 Epoch  90 Step:    33400 Batch Loss:     1.365443 Tokens per Sec:     7280, Lr: 0.000300\n",
            "2019-11-26 19:03:29,815 Epoch  90 Step:    33500 Batch Loss:     1.064060 Tokens per Sec:     7024, Lr: 0.000300\n",
            "2019-11-26 19:04:01,008 Epoch  90 Step:    33600 Batch Loss:     1.436276 Tokens per Sec:     7110, Lr: 0.000300\n",
            "2019-11-26 19:04:31,471 Epoch  90: total training loss 514.47\n",
            "2019-11-26 19:04:31,472 EPOCH 91\n",
            "2019-11-26 19:04:32,149 Epoch  91 Step:    33700 Batch Loss:     1.338671 Tokens per Sec:     6697, Lr: 0.000300\n",
            "2019-11-26 19:05:03,351 Epoch  91 Step:    33800 Batch Loss:     1.442129 Tokens per Sec:     7110, Lr: 0.000300\n",
            "2019-11-26 19:05:34,612 Epoch  91 Step:    33900 Batch Loss:     1.551265 Tokens per Sec:     7179, Lr: 0.000300\n",
            "2019-11-26 19:06:05,931 Epoch  91 Step:    34000 Batch Loss:     1.690044 Tokens per Sec:     7107, Lr: 0.000300\n",
            "2019-11-26 19:07:45,607 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 19:07:45,607 Saving new checkpoint.\n",
            "2019-11-26 19:07:45,927 Example #0\n",
            "2019-11-26 19:07:45,928 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 19:07:45,928 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 19:07:45,928 \tHypothesis: Bosenyi kgahlanong le bothata, go a lemogela gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ba bathobaso ka moka ka nageng.\n",
            "2019-11-26 19:07:45,928 Example #1\n",
            "2019-11-26 19:07:45,929 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 19:07:45,929 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 19:07:45,929 \tHypothesis: Kabinete e lebogišitše Proteas ge e le gore dikgopolo tše di bonagalago go la England mo bekeng ye e fetilego.\n",
            "2019-11-26 19:07:45,929 Example #2\n",
            "2019-11-26 19:07:45,930 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 19:07:45,930 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 19:07:45,930 \tHypothesis: Application for resution of South African citizenship, BI-175\n",
            "2019-11-26 19:07:45,930 Example #3\n",
            "2019-11-26 19:07:45,930 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 19:07:45,930 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 19:07:45,931 \tHypothesis: Pholisi ya rena ke go fa tshedimošo ge e ba gona.\n",
            "2019-11-26 19:07:45,931 Validation result (greedy) at epoch  91, step    34000: bleu:  18.77, loss: 69352.6328, ppl:   8.0906, duration: 99.9989s\n",
            "2019-11-26 19:08:08,254 Epoch  91: total training loss 511.12\n",
            "2019-11-26 19:08:08,254 EPOCH 92\n",
            "2019-11-26 19:08:16,886 Epoch  92 Step:    34100 Batch Loss:     0.589966 Tokens per Sec:     7059, Lr: 0.000300\n",
            "2019-11-26 19:08:48,113 Epoch  92 Step:    34200 Batch Loss:     1.666580 Tokens per Sec:     6937, Lr: 0.000300\n",
            "2019-11-26 19:09:19,578 Epoch  92 Step:    34300 Batch Loss:     1.370356 Tokens per Sec:     7203, Lr: 0.000300\n",
            "2019-11-26 19:09:50,725 Epoch  92 Step:    34400 Batch Loss:     1.529603 Tokens per Sec:     7085, Lr: 0.000300\n",
            "2019-11-26 19:10:05,826 Epoch  92: total training loss 513.05\n",
            "2019-11-26 19:10:05,826 EPOCH 93\n",
            "2019-11-26 19:10:21,793 Epoch  93 Step:    34500 Batch Loss:     1.132053 Tokens per Sec:     7145, Lr: 0.000300\n",
            "2019-11-26 19:10:52,947 Epoch  93 Step:    34600 Batch Loss:     1.275454 Tokens per Sec:     7075, Lr: 0.000300\n",
            "2019-11-26 19:11:24,273 Epoch  93 Step:    34700 Batch Loss:     1.204193 Tokens per Sec:     7018, Lr: 0.000300\n",
            "2019-11-26 19:11:54,946 Epoch  93 Step:    34800 Batch Loss:     1.160757 Tokens per Sec:     6928, Lr: 0.000300\n",
            "2019-11-26 19:12:03,700 Epoch  93: total training loss 511.93\n",
            "2019-11-26 19:12:03,700 EPOCH 94\n",
            "2019-11-26 19:12:26,025 Epoch  94 Step:    34900 Batch Loss:     1.271307 Tokens per Sec:     7131, Lr: 0.000300\n",
            "2019-11-26 19:12:57,291 Epoch  94 Step:    35000 Batch Loss:     1.456589 Tokens per Sec:     7204, Lr: 0.000300\n",
            "2019-11-26 19:13:28,213 Epoch  94 Step:    35100 Batch Loss:     1.755976 Tokens per Sec:     7034, Lr: 0.000300\n",
            "2019-11-26 19:13:59,315 Epoch  94 Step:    35200 Batch Loss:     1.093206 Tokens per Sec:     7088, Lr: 0.000300\n",
            "2019-11-26 19:14:00,183 Epoch  94: total training loss 505.33\n",
            "2019-11-26 19:14:00,183 EPOCH 95\n",
            "2019-11-26 19:14:30,513 Epoch  95 Step:    35300 Batch Loss:     1.600085 Tokens per Sec:     7121, Lr: 0.000300\n",
            "2019-11-26 19:15:01,781 Epoch  95 Step:    35400 Batch Loss:     1.425492 Tokens per Sec:     7098, Lr: 0.000300\n",
            "2019-11-26 19:15:32,869 Epoch  95 Step:    35500 Batch Loss:     1.330235 Tokens per Sec:     7158, Lr: 0.000300\n",
            "2019-11-26 19:15:57,033 Epoch  95: total training loss 502.70\n",
            "2019-11-26 19:15:57,033 EPOCH 96\n",
            "2019-11-26 19:16:03,879 Epoch  96 Step:    35600 Batch Loss:     0.905454 Tokens per Sec:     6839, Lr: 0.000300\n",
            "2019-11-26 19:16:35,100 Epoch  96 Step:    35700 Batch Loss:     1.564023 Tokens per Sec:     7116, Lr: 0.000300\n",
            "2019-11-26 19:17:05,982 Epoch  96 Step:    35800 Batch Loss:     1.571181 Tokens per Sec:     7113, Lr: 0.000300\n",
            "2019-11-26 19:17:36,985 Epoch  96 Step:    35900 Batch Loss:     1.630179 Tokens per Sec:     6972, Lr: 0.000300\n",
            "2019-11-26 19:17:54,153 Epoch  96: total training loss 502.13\n",
            "2019-11-26 19:17:54,153 EPOCH 97\n",
            "2019-11-26 19:18:08,219 Epoch  97 Step:    36000 Batch Loss:     1.427606 Tokens per Sec:     7130, Lr: 0.000300\n",
            "2019-11-26 19:19:47,967 Example #0\n",
            "2019-11-26 19:19:47,967 \tSource:     A crime against humanity, it is inevitable that the apartheid system will have had a detrimental effect on all black people in our country.\n",
            "2019-11-26 19:19:47,967 \tReference:  Bosenyi kgahlanong le botho, bo ka se efogwe gore tshepediša ya kgethollo e be e tlile go ba ya kotdi go batho baso ka moka ka nageng ye.\n",
            "2019-11-26 19:19:47,967 \tHypothesis: Bosenyi kgahlanong le bothata, go a lekalekana gore tshepedišo ya kgethologanyo e tla ba le seabe ka go batho ba bathobaso ka nageng.\n",
            "2019-11-26 19:19:47,967 Example #1\n",
            "2019-11-26 19:19:47,968 \tSource:     Cabinet also congratulated the Proteas for the spectacular series win against England last week.\n",
            "2019-11-26 19:19:47,968 \tReference:  Kabinete e lebogišitše di-Protea ka molokoloko wa papadi ya bona ye botse kgahlanong le naga ya England bekeng ya go feta.\n",
            "2019-11-26 19:19:47,968 \tHypothesis: Kabinete le yona e lebogile Proteas ka ge e le gore go na le dikgopolo tše di bonagalago tša go feta tše di lego gona kgahlanong le Seisimane le go feta.\n",
            "2019-11-26 19:19:47,968 Example #2\n",
            "2019-11-26 19:19:47,968 \tSource:     Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 19:19:47,968 \tReference:  Application for resumption of South African citizenship, BI-175\n",
            "2019-11-26 19:19:47,968 \tHypothesis: Kgopelo ya go hwetša gape maemo a boagi bja Afrika Borwa, BI-175\n",
            "2019-11-26 19:19:47,968 Example #3\n",
            "2019-11-26 19:19:47,968 \tSource:     Our policy is to provide information as and when it becomes available.\n",
            "2019-11-26 19:19:47,968 \tReference:  Pholisi ya rena ke go fana ka tshedimošo ge e eba gona.\n",
            "2019-11-26 19:19:47,969 \tHypothesis: Pholisi ya rena ke go fa tshedimošo ge e ba gona ge e le gona.\n",
            "2019-11-26 19:19:47,969 Validation result (greedy) at epoch  97, step    36000: bleu:  18.26, loss: 69699.4375, ppl:   8.1756, duration: 99.7494s\n",
            "2019-11-26 19:20:19,102 Epoch  97 Step:    36100 Batch Loss:     1.556029 Tokens per Sec:     7170, Lr: 0.000300\n",
            "2019-11-26 19:20:50,371 Epoch  97 Step:    36200 Batch Loss:     1.577519 Tokens per Sec:     7168, Lr: 0.000300\n",
            "2019-11-26 19:21:21,640 Epoch  97 Step:    36300 Batch Loss:     1.383021 Tokens per Sec:     7006, Lr: 0.000300\n",
            "2019-11-26 19:21:30,295 Epoch  97: total training loss 494.43\n",
            "2019-11-26 19:21:30,296 EPOCH 98\n",
            "2019-11-26 19:21:52,886 Epoch  98 Step:    36400 Batch Loss:     1.536853 Tokens per Sec:     7089, Lr: 0.000300\n",
            "2019-11-26 19:22:23,590 Epoch  98 Step:    36500 Batch Loss:     1.367542 Tokens per Sec:     6930, Lr: 0.000300\n",
            "2019-11-26 19:22:55,077 Epoch  98 Step:    36600 Batch Loss:     1.699567 Tokens per Sec:     7231, Lr: 0.000300\n",
            "2019-11-26 19:23:26,300 Epoch  98 Step:    36700 Batch Loss:     1.440331 Tokens per Sec:     7141, Lr: 0.000300\n",
            "2019-11-26 19:23:26,965 Epoch  98: total training loss 495.30\n",
            "2019-11-26 19:23:26,965 EPOCH 99\n",
            "2019-11-26 19:23:57,145 Epoch  99 Step:    36800 Batch Loss:     1.457906 Tokens per Sec:     6923, Lr: 0.000300\n",
            "2019-11-26 19:24:28,312 Epoch  99 Step:    36900 Batch Loss:     1.725332 Tokens per Sec:     7241, Lr: 0.000300\n",
            "2019-11-26 19:24:59,650 Epoch  99 Step:    37000 Batch Loss:     1.019154 Tokens per Sec:     7148, Lr: 0.000300\n",
            "2019-11-26 19:25:23,220 Epoch  99: total training loss 491.38\n",
            "2019-11-26 19:25:23,221 EPOCH 100\n",
            "2019-11-26 19:25:31,165 Epoch 100 Step:    37100 Batch Loss:     1.277079 Tokens per Sec:     7310, Lr: 0.000300\n",
            "2019-11-26 19:26:01,725 Epoch 100 Step:    37200 Batch Loss:     1.456917 Tokens per Sec:     6986, Lr: 0.000300\n",
            "2019-11-26 19:26:32,940 Epoch 100 Step:    37300 Batch Loss:     1.294251 Tokens per Sec:     7173, Lr: 0.000300\n",
            "2019-11-26 19:27:03,999 Epoch 100 Step:    37400 Batch Loss:     1.218265 Tokens per Sec:     7129, Lr: 0.000300\n",
            "2019-11-26 19:27:19,447 Epoch 100: total training loss 490.24\n",
            "2019-11-26 19:27:19,447 Training ended after 100 epochs.\n",
            "2019-11-26 19:27:19,447 Best validation result (greedy) at step    34000:   8.09 ppl.\n",
            "2019-11-26 19:28:33,199  dev bleu:  19.13 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 19:28:33,201 Translations saved to: models/ennso_transformer/00034000.hyps.dev\n",
            "2019-11-26 19:31:16,886 test bleu:   6.46 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 19:31:16,888 Translations saved to: models/ennso_transformer/00034000.hyps.test\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "outputId": "b115e6aa-184b-41c8-ef31-dd2a26736b88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# Copy the created models from the notebook storage to google drive for persistant storage \n",
        "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"/content/drive/My Drive/masakhane/en-nso-baseline/models/ennso_transformer/\""
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot create symbolic link '/content/drive/My Drive/masakhane/en-nso-baseline/models/ennso_transformer/best.ckpt': Function not implemented\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "outputId": "0d1d618f-6d64-405b-b704-b6c168af61fe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 319
        }
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 2000\tLoss: 117066.43750\tPPL: 34.09238\tbleu: 0.45997\tLR: 0.00030000\t*\n",
            "Steps: 4000\tLoss: 99850.73438\tPPL: 20.28924\tbleu: 2.02629\tLR: 0.00030000\t*\n",
            "Steps: 6000\tLoss: 90909.43750\tPPL: 15.49547\tbleu: 4.43105\tLR: 0.00030000\t*\n",
            "Steps: 8000\tLoss: 85468.15625\tPPL: 13.15123\tbleu: 7.01447\tLR: 0.00030000\t*\n",
            "Steps: 10000\tLoss: 81320.25000\tPPL: 11.60542\tbleu: 8.58762\tLR: 0.00030000\t*\n",
            "Steps: 12000\tLoss: 78477.86719\tPPL: 10.65241\tbleu: 10.28089\tLR: 0.00030000\t*\n",
            "Steps: 14000\tLoss: 76418.59375\tPPL: 10.01123\tbleu: 11.75510\tLR: 0.00030000\t*\n",
            "Steps: 16000\tLoss: 74479.33594\tPPL: 9.44275\tbleu: 13.04247\tLR: 0.00030000\t*\n",
            "Steps: 18000\tLoss: 73475.71875\tPPL: 9.16134\tbleu: 13.00069\tLR: 0.00030000\t*\n",
            "Steps: 20000\tLoss: 72637.54688\tPPL: 8.93275\tbleu: 14.77952\tLR: 0.00030000\t*\n",
            "Steps: 22000\tLoss: 71780.03906\tPPL: 8.70480\tbleu: 15.04306\tLR: 0.00030000\t*\n",
            "Steps: 24000\tLoss: 71055.67969\tPPL: 8.51678\tbleu: 16.53947\tLR: 0.00030000\t*\n",
            "Steps: 26000\tLoss: 70854.23438\tPPL: 8.46521\tbleu: 16.94294\tLR: 0.00030000\t*\n",
            "Steps: 28000\tLoss: 69797.82031\tPPL: 8.19987\tbleu: 18.12371\tLR: 0.00030000\t*\n",
            "Steps: 30000\tLoss: 70076.97656\tPPL: 8.26917\tbleu: 18.49001\tLR: 0.00030000\t\n",
            "Steps: 32000\tLoss: 69808.25000\tPPL: 8.20245\tbleu: 18.72158\tLR: 0.00030000\t\n",
            "Steps: 34000\tLoss: 69352.63281\tPPL: 8.09056\tbleu: 18.77358\tLR: 0.00030000\t*\n",
            "Steps: 36000\tLoss: 69699.43750\tPPL: 8.17559\tbleu: 18.25963\tLR: 0.00030000\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "outputId": "204d1778-99db-48d2-8af6-89053a46af53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-11-26 13:49:10,919 Hello! This is Joey-NMT.\n",
            "2019-11-26 13:49:45,068  dev bleu:  48.65 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 13:50:55,979 test bleu:  19.56 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k9QbsG8rCJBX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}