en-af/autshumato-baseline/enaf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "enaf.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "b1921b80-ca0f-4d61-d1b1-dd6f488877e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"af\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "8ab7ecfd-cbc3-4cae-ca91-aff9991f9bd8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-af-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "ebf2668b-ac76-4753-fd01-87d0bac351c0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 25.8MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 27.1MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 31.8MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 34.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 37.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 39.3MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 40.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 9.3MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "colab": {}
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "outputId": "ba03d09d-43bb-466e-9e26-c9149b89d1b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-11-26 11:15:28--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.005s  \n",
            "\n",
            "2019-11-26 11:15:29 (52.1 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2019-11-26 11:15:31--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-af.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 203446 (199K) [text/plain]\n",
            "Saving to: ‘test.en-af.en’\n",
            "\n",
            "test.en-af.en       100%[===================>] 198.68K  --.-KB/s    in 0.004s  \n",
            "\n",
            "2019-11-26 11:15:31 (53.0 MB/s) - ‘test.en-af.en’ saved [203446/203446]\n",
            "\n",
            "--2019-11-26 11:15:39--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-af.af\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 212484 (208K) [text/plain]\n",
            "Saving to: ‘test.en-af.af’\n",
            "\n",
            "test.en-af.af       100%[===================>] 207.50K  --.-KB/s    in 0.003s  \n",
            "\n",
            "2019-11-26 11:15:39 (65.1 MB/s) - ‘test.en-af.af’ saved [212484/212484]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "outputId": "0d978a04-cf67-494a-f582-28e365f6d3aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "8e58933f-2bfa-4417-d29f-0a70da4486ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'enaf_parallel.train.' + source_language\n",
        "target_file = 'enaf_parallel.train.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 1/50171 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>( 3 ) if material transfer agreements or benef...</td>\n",
              "      <td>( 3 ) indien materiaaloordragooreenkomste of v...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>( a ) signed material transfer agreements or b...</td>\n",
              "      <td>( a ) getekende materiaaloordragooreenkomste o...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>( b ) if it has not been possible to conclude ...</td>\n",
              "      <td>( b ) indien dit nie moontlik was om sodanige ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0  ( 3 ) if material transfer agreements or benef...  ( 3 ) indien materiaaloordragooreenkomste of v...\n",
              "1  ( a ) signed material transfer agreements or b...  ( a ) getekende materiaaloordragooreenkomste o...\n",
              "2  ( b ) if it has not been possible to conclude ...  ( b ) indien dit nie moontlik was om sodanige ..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "c43fbf86-b2e6-4136-f206-d57519166978",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "outputId": "7aeda118-931b-40d6-d0d9-9eb21236b4bd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.17.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 8.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (41.6.0)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144662 sha256=839189df0920c81aeedf1e6572a35a13de649e40fcf0d61080db2673c1e227b1\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n",
            "00:00:00.02 0.00 percent complete\n",
            "00:00:14.44 2.08 percent complete\n",
            "00:00:28.83 4.17 percent complete\n",
            "00:00:43.76 6.25 percent complete\n",
            "00:00:58.65 8.34 percent complete\n",
            "00:01:13.09 10.42 percent complete\n",
            "00:01:27.06 12.50 percent complete\n",
            "00:01:40.52 14.59 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: ':']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:01:54.51 16.67 percent complete\n",
            "00:02:08.22 18.75 percent complete\n",
            "00:02:21.96 20.84 percent complete\n",
            "00:02:36.32 22.92 percent complete\n",
            "00:02:50.34 25.01 percent complete\n",
            "00:03:04.63 27.09 percent complete\n",
            "00:03:18.58 29.17 percent complete\n",
            "00:03:32.73 31.26 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '|']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:03:46.90 33.34 percent complete\n",
            "00:04:01.33 35.42 percent complete\n",
            "00:04:15.57 37.51 percent complete\n",
            "00:04:29.17 39.59 percent complete\n",
            "00:04:42.97 41.68 percent complete\n",
            "00:04:57.24 43.76 percent complete\n",
            "00:05:11.76 45.84 percent complete\n",
            "00:05:25.78 47.93 percent complete\n",
            "00:05:39.29 50.01 percent complete\n",
            "00:05:53.25 52.09 percent complete\n",
            "00:06:07.30 54.18 percent complete\n",
            "00:06:21.92 56.26 percent complete\n",
            "00:06:35.96 58.35 percent complete\n",
            "00:06:50.12 60.43 percent complete\n",
            "00:07:04.53 62.51 percent complete\n",
            "00:07:18.47 64.60 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '/']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:07:32.24 66.68 percent complete\n",
            "00:07:46.07 68.76 percent complete\n",
            "00:07:59.82 70.85 percent complete\n",
            "00:08:13.33 72.93 percent complete\n",
            "00:08:26.94 75.02 percent complete\n",
            "00:08:41.12 77.10 percent complete\n",
            "00:08:55.04 79.18 percent complete\n",
            "00:09:09.58 81.27 percent complete\n",
            "00:09:23.75 83.35 percent complete\n",
            "00:09:37.88 85.43 percent complete\n",
            "00:09:52.05 87.52 percent complete\n",
            "00:10:06.92 89.60 percent complete\n",
            "00:10:21.68 91.69 percent complete\n",
            "00:10:35.72 93.77 percent complete\n",
            "00:10:50.09 95.85 percent complete\n",
            "00:11:04.32 97.94 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "outputId": "844bdd09-19b3-4e63-ccee-3c87644a2597",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 810
        }
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.af <==\n",
            "as persone aan ' n oortreding ingevolge dié verordening skuldig bevind word , welke oortreding skade aan of verlies van eiendom veroorsaak het en / of ' n nadelige uitwerking op die omgewing gehad het , kan die hof dan , benewens enige ander vonnis wat opgelê is , die volgende doen :\n",
            "* Small Claims Courts Act (Wet) (File type: pdf; size: 487 KB) (net in Engels beskikbaar)\n",
            "* Kinders wat min urineer\n",
            "Is Berading Nodig Voordat Mens 'n Vigstoets Ondergaan?\n",
            "* Mei/Junie 2006, Vol. 2, Issue 9\n",
            "Die regering moet seker maak dat mense toegang het tot behoorlike behuising.\n",
            "Dit is balju's se plig om alle mense, ryk en arm, te alle tye met empatie en waardigheid te behandel.\n",
            "Tuisblad > Jou Regering > Wes-Kaapse Provinsiale Regering > Wes-Kaap Onderwysdepartement > Fasiliteite > Sentrums vir Basiese Onderwys en Opleiding vir Volwassenes (BOOV), en Gemeenskapsleersentrums > Siyafuna Ukufunda Adult Learning Centre\n",
            "* Hoe om Aansoek te Doen om 'n Lisensie\n",
            "Vir gereelde ondersoeke en medikasie moet kliënte die kliniek besoek.\n",
            "\n",
            "==> train.en <==\n",
            "persons convicted of an offence under this bylaw which caused damage to or loss of property and / or which has had an adverse impact on the environment then , in addition to any other sentence imposed , the court may :\n",
            "* Small Claims Courts Act (Act) (File type: pdf; size: 487 KB)\n",
            "* Passing of a little urine\n",
            "Is Counselling Necessary before Having an Aids Test?\n",
            "* May/June 2006, Vol. 2, Issue 9\n",
            "The government must make sure people get access to proper housing.\n",
            "It is the duty of sheriffs to serve all people, rich and poor, with empathy and dignity at all times.\n",
            "Home > Your Government > Provincial Government of the Western Cape > Western Cape Education Department > Facilities > ABET & Community Learning Centres > Siyafuna Ukufunda Adult Learning Centre\n",
            "* How to Apply for a Licence\n",
            "Clients must return to the clinic for regular check-ups and medication.\n",
            "==> dev.af <==\n",
            "GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "Die Drostdy\n",
            "die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "* indiwidue se private optrede\n",
            "en die bepaling van enige diens deur die Raad ;\n",
            "Hulle het egter dieselfde magte en pligte as enige polisielid terwyl hulle aan diens is.\n",
            "Wanneer mens hoes of nies, moet enige mens, veral dié met TB of MMW TB hulle neus en mond met 'n snesie bedek om die verspreiding van basille deur die lug te keer.\n",
            "Approval for a boar for artificial insemination\n",
            "ek voel ontsaglik trots dat demokratiese suid-afrika verstanding en fyngevoelig genoeg was om erkenning te gee aan wat albert luthuli en oliver tambo vir ons volk beteken deur twee van ons nasionale ordes na hulle te vernoem : die orde van luthuli , en die orde van die metgeselle van or tambo .\n",
            "Indien u bestuurslisensie verlore raak, of gesteel of vernietig word, kan u om vervanging daarvan aansoek doen.\n",
            "\n",
            "==> dev.en <==\n",
            "GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "The Drostdy\n",
            "the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "* private acts by individuals\n",
            "and the provision of any service by the Council ;\n",
            "However, they have the same powers and duties as any police officer while they are on duty.\n",
            "When coughing or sneezing, any person and especially those with TB or MDR TB should cover the nose and mouth with a tissue to prevent air-borne spread of the bacilli.\n",
            "Approval for a boar for artificial insemination\n",
            "i feel immensely proud that democratic south africa has had the sense and sensitivity to acknowledge what albert luthuli and oliver tambo mean to our nation by naming two of our national orders after them - the order of luthuli , and the order of the companions of o.r .\n",
            "If your driving licence is lost, stolen or destroyed, you can apply for a replacement licence.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "outputId": "17c99725-9750-46ca-9b40-f39142d2abbc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 15, done.\u001b[K\n",
            "remote: Counting objects: 100% (15/15), done.\u001b[K\n",
            "remote: Compressing objects: 100% (12/12), done.\u001b[K\n",
            "remote: Total 2199 (delta 4), reused 5 (delta 3), pack-reused 2184\u001b[K\n",
            "Receiving objects: 100% (2199/2199), 2.60 MiB | 963.00 KiB/s, done.\n",
            "Resolving deltas: 100% (1525/1525), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (4.3.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.4)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (41.6.0)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "  Downloading https://files.pythonhosted.org/packages/0e/e5/93d252182f7cbd4b59bb3ec5797e2ce33cfd6f5aadaf327db170cf4b7887/sacrebleu-1.4.2-py3-none-any.whl\n",
            "Collecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)\n",
            "\u001b[K     |████████████████████████████████| 266kB 39.2MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
            "\u001b[K     |████████████████████████████████| 307kB 52.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->joeynmt==0.0.1) (0.46)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.0)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.5)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
            "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.3.2)\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 9.7MB/s \n",
            "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Collecting astroid<2.4,>=2.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 47.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.9.11)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/31/d3/9d1802c161626d0278bafb1ffb32f76b9d01e123881bbf9d91e8ccf28e18/typed_ast-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (736kB)\n",
            "\u001b[K     |████████████████████████████████| 737kB 52.4MB/s \n",
            "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 9.4MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=724dda0553f12fc3e25ad81a5090e464a3502c77c6529930463c0eba6d4f5066\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-ipbh1mlc/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=1db864996a0509685026d3509c0b3476b62cdc0bd8ce97860fc2c6e1d06661f9\n",
            "  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030\n",
            "Successfully built joeynmt pyyaml\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, isort, mccabe, typed-ast, lazy-object-proxy, astroid, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.1.2 sacrebleu-1.4.2 subword-nmt-0.3.7 typed-ast-1.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "outputId": "b9f3ebf3-3345-44e9-853c-9e0fc9aa8cec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        }
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.4000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Afrikaans Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.bpe.en  test.bpe.af  test.en-any.en  train.bpe.en\n",
            "dev.af\t\tdev.en\t    test.bpe.en  train.af\t train.en\n",
            "dev.bpe.af\ttest.af     test.en\t train.bpe.af\n",
            "bpe.codes.4000\tdev.bpe.en  test.bpe.af  test.en-any.en  train.bpe.en\n",
            "dev.af\t\tdev.en\t    test.bpe.en  train.af\t train.en\n",
            "dev.bpe.af\ttest.af     test.en\t train.bpe.af\n",
            "BPE Afrikaans Sentences\n",
            "E@@ k mo@@ es h@@ om om@@ k@@ oop sodat hy sy o@@ ë s@@ ou sluit vir die maatskappy se on@@ eer@@ likheid .\n",
            "Ge@@ vol@@ g@@ lik was ek beken@@ d as ’ n on@@ eer@@ like man . To@@ e ek in die waar@@ heid kom , het ek gew@@ ei@@ er om daar@@ me@@ e voor@@ t te gaan , al het ek ’ n go@@ eie sal@@ aris ver@@ dien .\n",
            "E@@ k stel ’ n go@@ eie voorbe@@ eld vir my twee se@@ un@@ s , en ek het voor@@ regte in die gemeen@@ te ge@@ kry .\n",
            "Bel@@ ast@@ ing@@ ou@@ di@@ te@@ urs en ander met wie ek besi@@ gheid doen , we@@ et n@@ ou dat ek ’ n eer@@ like man is . ”\n",
            "R@@ ut het na I@@ s@@ ra@@ el ge@@ trek , waar sy die w@@ are G@@ o@@ d k@@ on aan@@ bi@@ d .\n",
            "Combined BPE Vocab\n",
            "Ö@@\n",
            "«@@\n",
            "(Servic@@\n",
            "Ô@@\n",
            "onomiese\n",
            "GOVER@@\n",
            "&nbsp;\n",
            "softw@@\n",
            "Ã@@\n",
            "―@@\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlMitUHR8Qy-",
        "outputId": "6ab7079a-c488-4e18-a2ab-8f91792f0bc3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\""
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.bpe.en  test.bpe.af  test.en-any.en  train.bpe.en\n",
            "dev.af\t\tdev.en\t    test.bpe.en  train.af\t train.en\n",
            "dev.bpe.af\ttest.af     test.en\t train.bpe.af\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"data/{name}/train.bpe\"\n",
        "    dev:   \"data/{name}/dev.bpe\"\n",
        "    test:  \"data/{name}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"data/{name}/vocab.txt\"\n",
        "    trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 100                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 2000          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "outputId": "dfc0c383-dc2b-4389-c33d-36863014b6ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-11-26 11:31:47,451 Hello! This is Joey-NMT.\n",
            "2019-11-26 11:31:49,152 Total params: 12138240\n",
            "2019-11-26 11:31:49,154 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2019-11-26 11:31:56,433 cfg.name                           : enaf_transformer\n",
            "2019-11-26 11:31:56,433 cfg.data.src                       : en\n",
            "2019-11-26 11:31:56,434 cfg.data.trg                       : af\n",
            "2019-11-26 11:31:56,434 cfg.data.train                     : data/enaf/train.bpe\n",
            "2019-11-26 11:31:56,434 cfg.data.dev                       : data/enaf/dev.bpe\n",
            "2019-11-26 11:31:56,434 cfg.data.test                      : data/enaf/test.bpe\n",
            "2019-11-26 11:31:56,434 cfg.data.level                     : bpe\n",
            "2019-11-26 11:31:56,434 cfg.data.lowercase                 : False\n",
            "2019-11-26 11:31:56,434 cfg.data.max_sent_length           : 100\n",
            "2019-11-26 11:31:56,434 cfg.data.src_vocab                 : data/enaf/vocab.txt\n",
            "2019-11-26 11:31:56,434 cfg.data.trg_vocab                 : data/enaf/vocab.txt\n",
            "2019-11-26 11:31:56,434 cfg.testing.beam_size              : 5\n",
            "2019-11-26 11:31:56,434 cfg.testing.alpha                  : 1.0\n",
            "2019-11-26 11:31:56,434 cfg.training.random_seed           : 42\n",
            "2019-11-26 11:31:56,434 cfg.training.optimizer             : adam\n",
            "2019-11-26 11:31:56,434 cfg.training.normalization         : tokens\n",
            "2019-11-26 11:31:56,434 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2019-11-26 11:31:56,434 cfg.training.scheduling            : plateau\n",
            "2019-11-26 11:31:56,434 cfg.training.patience              : 5\n",
            "2019-11-26 11:31:56,434 cfg.training.learning_rate_factor  : 0.5\n",
            "2019-11-26 11:31:56,434 cfg.training.learning_rate_warmup  : 1000\n",
            "2019-11-26 11:31:56,434 cfg.training.decrease_factor       : 0.7\n",
            "2019-11-26 11:31:56,434 cfg.training.loss                  : crossentropy\n",
            "2019-11-26 11:31:56,434 cfg.training.learning_rate         : 0.0003\n",
            "2019-11-26 11:31:56,434 cfg.training.learning_rate_min     : 1e-08\n",
            "2019-11-26 11:31:56,434 cfg.training.weight_decay          : 0.0\n",
            "2019-11-26 11:31:56,434 cfg.training.label_smoothing       : 0.1\n",
            "2019-11-26 11:31:56,435 cfg.training.batch_size            : 4096\n",
            "2019-11-26 11:31:56,435 cfg.training.batch_type            : token\n",
            "2019-11-26 11:31:56,435 cfg.training.eval_batch_size       : 3600\n",
            "2019-11-26 11:31:56,435 cfg.training.eval_batch_type       : token\n",
            "2019-11-26 11:31:56,435 cfg.training.batch_multiplier      : 1\n",
            "2019-11-26 11:31:56,435 cfg.training.early_stopping_metric : ppl\n",
            "2019-11-26 11:31:56,435 cfg.training.epochs                : 100\n",
            "2019-11-26 11:31:56,435 cfg.training.validation_freq       : 2000\n",
            "2019-11-26 11:31:56,435 cfg.training.logging_freq          : 100\n",
            "2019-11-26 11:31:56,435 cfg.training.eval_metric           : bleu\n",
            "2019-11-26 11:31:56,435 cfg.training.model_dir             : models/enaf_transformer\n",
            "2019-11-26 11:31:56,435 cfg.training.overwrite             : False\n",
            "2019-11-26 11:31:56,435 cfg.training.shuffle               : True\n",
            "2019-11-26 11:31:56,435 cfg.training.use_cuda              : True\n",
            "2019-11-26 11:31:56,435 cfg.training.max_output_length     : 100\n",
            "2019-11-26 11:31:56,435 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2019-11-26 11:31:56,435 cfg.training.keep_last_ckpts       : 3\n",
            "2019-11-26 11:31:56,435 cfg.model.initializer              : xavier\n",
            "2019-11-26 11:31:56,435 cfg.model.bias_initializer         : zeros\n",
            "2019-11-26 11:31:56,435 cfg.model.init_gain                : 1.0\n",
            "2019-11-26 11:31:56,435 cfg.model.embed_initializer        : xavier\n",
            "2019-11-26 11:31:56,435 cfg.model.embed_init_gain          : 1.0\n",
            "2019-11-26 11:31:56,435 cfg.model.tied_embeddings          : True\n",
            "2019-11-26 11:31:56,435 cfg.model.tied_softmax             : True\n",
            "2019-11-26 11:31:56,435 cfg.model.encoder.type             : transformer\n",
            "2019-11-26 11:31:56,435 cfg.model.encoder.num_layers       : 6\n",
            "2019-11-26 11:31:56,435 cfg.model.encoder.num_heads        : 4\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.embeddings.scale : True\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.hidden_size      : 256\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.ff_size          : 1024\n",
            "2019-11-26 11:31:56,436 cfg.model.encoder.dropout          : 0.3\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.type             : transformer\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.num_layers       : 6\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.num_heads        : 4\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.embeddings.scale : True\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.hidden_size      : 256\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.ff_size          : 1024\n",
            "2019-11-26 11:31:56,436 cfg.model.decoder.dropout          : 0.3\n",
            "2019-11-26 11:31:56,436 Data set sizes: \n",
            "\ttrain 46442,\n",
            "\tvalid 1000,\n",
            "\ttest 2682\n",
            "2019-11-26 11:31:56,436 First training example:\n",
            "\t[SRC] persons con@@ vic@@ ted of an off@@ ence under this by@@ law which ca@@ used da@@ ma@@ ge to or lo@@ ss of property and / or which has had an ad@@ ver@@ se imp@@ act on the environ@@ ment then , in addi@@ tion to any other s@@ ent@@ ence im@@ posed , the court may :\n",
            "\t[TRG] as persone aan ' n oortre@@ ding ingevolge dié ver@@ or@@ den@@ ing sk@@ ul@@ dig be@@ vind word , wel@@ ke oortre@@ ding ska@@ de aan of ver@@ lies van eiendom veroor@@ saak het en / of ' n n@@ ad@@ el@@ ige uit@@ werking op die omgewing ge@@ had het , kan die hof dan , bene@@ w@@ ens enige ander v@@ on@@ n@@ is wat op@@ gelê is , die volgende doen :\n",
            "2019-11-26 11:31:56,436 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) > (5) die (6) the (7) of (8) , (9) van\n",
            "2019-11-26 11:31:56,437 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) > (5) die (6) the (7) of (8) , (9) van\n",
            "2019-11-26 11:31:56,437 Number of Src words (types): 4211\n",
            "2019-11-26 11:31:56,438 Number of Trg words (types): 4211\n",
            "2019-11-26 11:31:56,438 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4211),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4211))\n",
            "2019-11-26 11:31:56,442 EPOCH 1\n",
            "2019-11-26 11:32:11,765 Epoch   1 Step:      100 Batch Loss:     5.608022 Tokens per Sec:    15825, Lr: 0.000300\n",
            "2019-11-26 11:32:27,294 Epoch   1 Step:      200 Batch Loss:     5.463318 Tokens per Sec:    16006, Lr: 0.000300\n",
            "2019-11-26 11:32:42,783 Epoch   1 Step:      300 Batch Loss:     5.370947 Tokens per Sec:    15892, Lr: 0.000300\n",
            "2019-11-26 11:32:58,332 Epoch   1 Step:      400 Batch Loss:     4.917095 Tokens per Sec:    15679, Lr: 0.000300\n",
            "2019-11-26 11:33:04,100 Epoch   1: total training loss 2422.10\n",
            "2019-11-26 11:33:04,101 EPOCH 2\n",
            "2019-11-26 11:33:14,286 Epoch   2 Step:      500 Batch Loss:     4.905149 Tokens per Sec:    15494, Lr: 0.000300\n",
            "2019-11-26 11:33:30,507 Epoch   2 Step:      600 Batch Loss:     4.610600 Tokens per Sec:    15285, Lr: 0.000300\n",
            "2019-11-26 11:33:46,814 Epoch   2 Step:      700 Batch Loss:     4.996385 Tokens per Sec:    15002, Lr: 0.000300\n",
            "2019-11-26 11:34:03,302 Epoch   2 Step:      800 Batch Loss:     4.187446 Tokens per Sec:    14752, Lr: 0.000300\n",
            "2019-11-26 11:34:14,964 Epoch   2: total training loss 2021.33\n",
            "2019-11-26 11:34:14,965 EPOCH 3\n",
            "2019-11-26 11:34:19,537 Epoch   3 Step:      900 Batch Loss:     3.939539 Tokens per Sec:    14351, Lr: 0.000300\n",
            "2019-11-26 11:34:35,657 Epoch   3 Step:     1000 Batch Loss:     3.818528 Tokens per Sec:    15270, Lr: 0.000300\n",
            "2019-11-26 11:34:51,732 Epoch   3 Step:     1100 Batch Loss:     3.625695 Tokens per Sec:    15168, Lr: 0.000300\n",
            "2019-11-26 11:35:07,852 Epoch   3 Step:     1200 Batch Loss:     3.946392 Tokens per Sec:    14928, Lr: 0.000300\n",
            "2019-11-26 11:35:24,259 Epoch   3 Step:     1300 Batch Loss:     3.731726 Tokens per Sec:    15277, Lr: 0.000300\n",
            "2019-11-26 11:35:25,934 Epoch   3: total training loss 1791.17\n",
            "2019-11-26 11:35:25,934 EPOCH 4\n",
            "2019-11-26 11:35:40,879 Epoch   4 Step:     1400 Batch Loss:     3.751532 Tokens per Sec:    15160, Lr: 0.000300\n",
            "2019-11-26 11:35:57,097 Epoch   4 Step:     1500 Batch Loss:     3.689330 Tokens per Sec:    15015, Lr: 0.000300\n",
            "2019-11-26 11:36:13,120 Epoch   4 Step:     1600 Batch Loss:     3.929770 Tokens per Sec:    14982, Lr: 0.000300\n",
            "2019-11-26 11:36:29,465 Epoch   4 Step:     1700 Batch Loss:     3.113327 Tokens per Sec:    15081, Lr: 0.000300\n",
            "2019-11-26 11:36:36,897 Epoch   4: total training loss 1576.17\n",
            "2019-11-26 11:36:36,898 EPOCH 5\n",
            "2019-11-26 11:36:45,780 Epoch   5 Step:     1800 Batch Loss:     3.865883 Tokens per Sec:    14533, Lr: 0.000300\n",
            "2019-11-26 11:37:02,071 Epoch   5 Step:     1900 Batch Loss:     4.273342 Tokens per Sec:    15221, Lr: 0.000300\n",
            "2019-11-26 11:37:18,372 Epoch   5 Step:     2000 Batch Loss:     3.196836 Tokens per Sec:    15284, Lr: 0.000300\n",
            "2019-11-26 11:38:07,776 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 11:38:07,776 Saving new checkpoint.\n",
            "2019-11-26 11:38:08,090 Example #0\n",
            "2019-11-26 11:38:08,090 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 11:38:08,090 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:38:08,090 \tHypothesis: GEWINGMUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:38:08,091 Example #1\n",
            "2019-11-26 11:38:08,091 \tSource:     The Drostdy\n",
            "2019-11-26 11:38:08,091 \tReference:  Die Drostdy\n",
            "2019-11-26 11:38:08,091 \tHypothesis: Die Drotale\n",
            "2019-11-26 11:38:08,091 Example #2\n",
            "2019-11-26 11:38:08,091 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 11:38:08,091 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 11:38:08,092 \tHypothesis: die raad van die verrigtinge van die verrigting van die verrigtinge van die verwante stel om die staatsdiensdiensneming van die ontwikkeling van die provinsie te verseker dat die provinsie en die ontwikkeling van die provinsie .\n",
            "2019-11-26 11:38:08,092 Example #3\n",
            "2019-11-26 11:38:08,092 \tSource:     * private acts by individuals\n",
            "2019-11-26 11:38:08,092 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 11:38:08,092 \tHypothesis: * openbare liggaam deur die uitbreiding\n",
            "2019-11-26 11:38:08,092 Validation result (greedy) at epoch   5, step     2000: bleu:  12.36, loss: 74854.2578, ppl:  24.7317, duration: 49.7199s\n",
            "2019-11-26 11:38:24,497 Epoch   5 Step:     2100 Batch Loss:     3.192021 Tokens per Sec:    15186, Lr: 0.000300\n",
            "2019-11-26 11:38:37,659 Epoch   5: total training loss 1406.83\n",
            "2019-11-26 11:38:37,659 EPOCH 6\n",
            "2019-11-26 11:38:40,691 Epoch   6 Step:     2200 Batch Loss:     2.781468 Tokens per Sec:    14106, Lr: 0.000300\n",
            "2019-11-26 11:38:56,794 Epoch   6 Step:     2300 Batch Loss:     3.204317 Tokens per Sec:    14912, Lr: 0.000300\n",
            "2019-11-26 11:39:13,039 Epoch   6 Step:     2400 Batch Loss:     3.285748 Tokens per Sec:    14922, Lr: 0.000300\n",
            "2019-11-26 11:39:29,271 Epoch   6 Step:     2500 Batch Loss:     2.839251 Tokens per Sec:    15336, Lr: 0.000300\n",
            "2019-11-26 11:39:45,650 Epoch   6 Step:     2600 Batch Loss:     3.585898 Tokens per Sec:    15151, Lr: 0.000300\n",
            "2019-11-26 11:39:49,013 Epoch   6: total training loss 1272.56\n",
            "2019-11-26 11:39:49,013 EPOCH 7\n",
            "2019-11-26 11:40:01,996 Epoch   7 Step:     2700 Batch Loss:     2.570757 Tokens per Sec:    15140, Lr: 0.000300\n",
            "2019-11-26 11:40:18,041 Epoch   7 Step:     2800 Batch Loss:     2.637045 Tokens per Sec:    15093, Lr: 0.000300\n",
            "2019-11-26 11:40:34,218 Epoch   7 Step:     2900 Batch Loss:     3.006508 Tokens per Sec:    15137, Lr: 0.000300\n",
            "2019-11-26 11:40:50,598 Epoch   7 Step:     3000 Batch Loss:     2.436803 Tokens per Sec:    15142, Lr: 0.000300\n",
            "2019-11-26 11:41:00,100 Epoch   7: total training loss 1166.51\n",
            "2019-11-26 11:41:00,100 EPOCH 8\n",
            "2019-11-26 11:41:06,840 Epoch   8 Step:     3100 Batch Loss:     2.900514 Tokens per Sec:    14948, Lr: 0.000300\n",
            "2019-11-26 11:41:22,984 Epoch   8 Step:     3200 Batch Loss:     2.230870 Tokens per Sec:    15114, Lr: 0.000300\n",
            "2019-11-26 11:41:39,189 Epoch   8 Step:     3300 Batch Loss:     2.176509 Tokens per Sec:    15150, Lr: 0.000300\n",
            "2019-11-26 11:41:55,429 Epoch   8 Step:     3400 Batch Loss:     2.247450 Tokens per Sec:    15147, Lr: 0.000300\n",
            "2019-11-26 11:42:10,746 Epoch   8: total training loss 1078.80\n",
            "2019-11-26 11:42:10,746 EPOCH 9\n",
            "2019-11-26 11:42:11,815 Epoch   9 Step:     3500 Batch Loss:     1.725628 Tokens per Sec:    14151, Lr: 0.000300\n",
            "2019-11-26 11:42:27,897 Epoch   9 Step:     3600 Batch Loss:     1.688234 Tokens per Sec:    15190, Lr: 0.000300\n",
            "2019-11-26 11:42:44,198 Epoch   9 Step:     3700 Batch Loss:     2.664847 Tokens per Sec:    15236, Lr: 0.000300\n",
            "2019-11-26 11:43:00,309 Epoch   9 Step:     3800 Batch Loss:     1.284317 Tokens per Sec:    14963, Lr: 0.000300\n",
            "2019-11-26 11:43:16,551 Epoch   9 Step:     3900 Batch Loss:     2.050797 Tokens per Sec:    15223, Lr: 0.000300\n",
            "2019-11-26 11:43:21,720 Epoch   9: total training loss 1023.69\n",
            "2019-11-26 11:43:21,720 EPOCH 10\n",
            "2019-11-26 11:43:32,900 Epoch  10 Step:     4000 Batch Loss:     3.058451 Tokens per Sec:    15199, Lr: 0.000300\n",
            "2019-11-26 11:44:22,239 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 11:44:22,240 Saving new checkpoint.\n",
            "2019-11-26 11:44:22,499 Example #0\n",
            "2019-11-26 11:44:22,500 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 11:44:22,500 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:44:22,500 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:44:22,500 Example #1\n",
            "2019-11-26 11:44:22,500 \tSource:     The Drostdy\n",
            "2019-11-26 11:44:22,500 \tReference:  Die Drostdy\n",
            "2019-11-26 11:44:22,500 \tHypothesis: Die Drostdy\n",
            "2019-11-26 11:44:22,500 Example #2\n",
            "2019-11-26 11:44:22,501 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 11:44:22,501 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 11:44:22,501 \tHypothesis: die voorwerp van die seksuele kategorie van komitees is om die uitvoer van persone te verander , en om die kommissie te verseker dat die kommissie se aktiwiteite in die kommissie se aktiwiteite .\n",
            "2019-11-26 11:44:22,501 Example #3\n",
            "2019-11-26 11:44:22,501 \tSource:     * private acts by individuals\n",
            "2019-11-26 11:44:22,501 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 11:44:22,501 \tHypothesis: * privaatliggame deur individuele\n",
            "2019-11-26 11:44:22,501 Validation result (greedy) at epoch  10, step     4000: bleu:  26.51, loss: 52469.3672, ppl:   9.4756, duration: 49.6011s\n",
            "2019-11-26 11:44:38,865 Epoch  10 Step:     4100 Batch Loss:     2.862207 Tokens per Sec:    15103, Lr: 0.000300\n",
            "2019-11-26 11:44:55,167 Epoch  10 Step:     4200 Batch Loss:     1.729798 Tokens per Sec:    15214, Lr: 0.000300\n",
            "2019-11-26 11:45:11,422 Epoch  10 Step:     4300 Batch Loss:     2.381748 Tokens per Sec:    15342, Lr: 0.000300\n",
            "2019-11-26 11:45:22,012 Epoch  10: total training loss 949.79\n",
            "2019-11-26 11:45:22,013 EPOCH 11\n",
            "2019-11-26 11:45:27,711 Epoch  11 Step:     4400 Batch Loss:     2.298461 Tokens per Sec:    14736, Lr: 0.000300\n",
            "2019-11-26 11:45:43,984 Epoch  11 Step:     4500 Batch Loss:     3.077555 Tokens per Sec:    14930, Lr: 0.000300\n",
            "2019-11-26 11:46:00,210 Epoch  11 Step:     4600 Batch Loss:     2.786949 Tokens per Sec:    14965, Lr: 0.000300\n",
            "2019-11-26 11:46:16,368 Epoch  11 Step:     4700 Batch Loss:     2.088893 Tokens per Sec:    15158, Lr: 0.000300\n",
            "2019-11-26 11:46:32,480 Epoch  11 Step:     4800 Batch Loss:     2.518680 Tokens per Sec:    15061, Lr: 0.000300\n",
            "2019-11-26 11:46:33,345 Epoch  11: total training loss 920.89\n",
            "2019-11-26 11:46:33,345 EPOCH 12\n",
            "2019-11-26 11:46:49,062 Epoch  12 Step:     4900 Batch Loss:     1.790246 Tokens per Sec:    15327, Lr: 0.000300\n",
            "2019-11-26 11:47:05,025 Epoch  12 Step:     5000 Batch Loss:     2.516450 Tokens per Sec:    14710, Lr: 0.000300\n",
            "2019-11-26 11:47:21,147 Epoch  12 Step:     5100 Batch Loss:     0.975306 Tokens per Sec:    15034, Lr: 0.000300\n",
            "2019-11-26 11:47:37,640 Epoch  12 Step:     5200 Batch Loss:     1.202131 Tokens per Sec:    15535, Lr: 0.000300\n",
            "2019-11-26 11:47:44,228 Epoch  12: total training loss 867.65\n",
            "2019-11-26 11:47:44,228 EPOCH 13\n",
            "2019-11-26 11:47:53,959 Epoch  13 Step:     5300 Batch Loss:     1.634143 Tokens per Sec:    15130, Lr: 0.000300\n",
            "2019-11-26 11:48:10,070 Epoch  13 Step:     5400 Batch Loss:     2.598377 Tokens per Sec:    15112, Lr: 0.000300\n",
            "2019-11-26 11:48:26,292 Epoch  13 Step:     5500 Batch Loss:     1.726901 Tokens per Sec:    14871, Lr: 0.000300\n",
            "2019-11-26 11:48:42,694 Epoch  13 Step:     5600 Batch Loss:     1.384664 Tokens per Sec:    15168, Lr: 0.000300\n",
            "2019-11-26 11:48:55,197 Epoch  13: total training loss 825.53\n",
            "2019-11-26 11:48:55,197 EPOCH 14\n",
            "2019-11-26 11:48:59,179 Epoch  14 Step:     5700 Batch Loss:     2.271524 Tokens per Sec:    15404, Lr: 0.000300\n",
            "2019-11-26 11:49:15,473 Epoch  14 Step:     5800 Batch Loss:     2.350677 Tokens per Sec:    14999, Lr: 0.000300\n",
            "2019-11-26 11:49:31,751 Epoch  14 Step:     5900 Batch Loss:     1.843396 Tokens per Sec:    15136, Lr: 0.000300\n",
            "2019-11-26 11:49:48,095 Epoch  14 Step:     6000 Batch Loss:     1.916567 Tokens per Sec:    15303, Lr: 0.000300\n",
            "2019-11-26 11:50:37,391 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 11:50:37,391 Saving new checkpoint.\n",
            "2019-11-26 11:50:37,650 Example #0\n",
            "2019-11-26 11:50:37,650 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 11:50:37,650 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:50:37,650 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:50:37,650 Example #1\n",
            "2019-11-26 11:50:37,651 \tSource:     The Drostdy\n",
            "2019-11-26 11:50:37,651 \tReference:  Die Drostdy\n",
            "2019-11-26 11:50:37,651 \tHypothesis: Die Drostdy\n",
            "2019-11-26 11:50:37,651 Example #2\n",
            "2019-11-26 11:50:37,651 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 11:50:37,651 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 11:50:37,651 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitoefening van persone te maak van die kommissie en te verseker dat die kommissie en betrokkenheid in die kommissie van die kommissie van die kommissie .\n",
            "2019-11-26 11:50:37,651 Example #3\n",
            "2019-11-26 11:50:37,652 \tSource:     * private acts by individuals\n",
            "2019-11-26 11:50:37,652 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 11:50:37,652 \tHypothesis: * private aksies deur individue\n",
            "2019-11-26 11:50:37,652 Validation result (greedy) at epoch  14, step     6000: bleu:  33.56, loss: 42362.4609, ppl:   6.1445, duration: 49.5567s\n",
            "2019-11-26 11:50:53,800 Epoch  14 Step:     6100 Batch Loss:     1.992535 Tokens per Sec:    15073, Lr: 0.000300\n",
            "2019-11-26 11:50:55,479 Epoch  14: total training loss 788.52\n",
            "2019-11-26 11:50:55,479 EPOCH 15\n",
            "2019-11-26 11:51:10,277 Epoch  15 Step:     6200 Batch Loss:     2.023537 Tokens per Sec:    15114, Lr: 0.000300\n",
            "2019-11-26 11:51:26,329 Epoch  15 Step:     6300 Batch Loss:     2.314336 Tokens per Sec:    15020, Lr: 0.000300\n",
            "2019-11-26 11:51:42,486 Epoch  15 Step:     6400 Batch Loss:     1.569038 Tokens per Sec:    15108, Lr: 0.000300\n",
            "2019-11-26 11:51:58,661 Epoch  15 Step:     6500 Batch Loss:     2.190273 Tokens per Sec:    15237, Lr: 0.000300\n",
            "2019-11-26 11:52:06,500 Epoch  15: total training loss 768.51\n",
            "2019-11-26 11:52:06,500 EPOCH 16\n",
            "2019-11-26 11:52:14,939 Epoch  16 Step:     6600 Batch Loss:     1.241720 Tokens per Sec:    15387, Lr: 0.000300\n",
            "2019-11-26 11:52:30,968 Epoch  16 Step:     6700 Batch Loss:     1.467147 Tokens per Sec:    14874, Lr: 0.000300\n",
            "2019-11-26 11:52:47,070 Epoch  16 Step:     6800 Batch Loss:     1.967465 Tokens per Sec:    15264, Lr: 0.000300\n",
            "2019-11-26 11:53:03,349 Epoch  16 Step:     6900 Batch Loss:     1.676328 Tokens per Sec:    15298, Lr: 0.000300\n",
            "2019-11-26 11:53:17,223 Epoch  16: total training loss 738.36\n",
            "2019-11-26 11:53:17,224 EPOCH 17\n",
            "2019-11-26 11:53:19,437 Epoch  17 Step:     7000 Batch Loss:     0.934059 Tokens per Sec:    13372, Lr: 0.000300\n",
            "2019-11-26 11:53:35,536 Epoch  17 Step:     7100 Batch Loss:     1.657806 Tokens per Sec:    14943, Lr: 0.000300\n",
            "2019-11-26 11:53:51,669 Epoch  17 Step:     7200 Batch Loss:     1.226249 Tokens per Sec:    15264, Lr: 0.000300\n",
            "2019-11-26 11:54:07,998 Epoch  17 Step:     7300 Batch Loss:     2.209456 Tokens per Sec:    15441, Lr: 0.000300\n",
            "2019-11-26 11:54:24,231 Epoch  17 Step:     7400 Batch Loss:     1.720668 Tokens per Sec:    15393, Lr: 0.000300\n",
            "2019-11-26 11:54:27,792 Epoch  17: total training loss 714.93\n",
            "2019-11-26 11:54:27,792 EPOCH 18\n",
            "2019-11-26 11:54:40,204 Epoch  18 Step:     7500 Batch Loss:     1.135857 Tokens per Sec:    14700, Lr: 0.000300\n",
            "2019-11-26 11:54:56,583 Epoch  18 Step:     7600 Batch Loss:     2.273018 Tokens per Sec:    15302, Lr: 0.000300\n",
            "2019-11-26 11:55:12,880 Epoch  18 Step:     7700 Batch Loss:     1.397834 Tokens per Sec:    15419, Lr: 0.000300\n",
            "2019-11-26 11:55:28,941 Epoch  18 Step:     7800 Batch Loss:     2.175472 Tokens per Sec:    15108, Lr: 0.000300\n",
            "2019-11-26 11:55:38,522 Epoch  18: total training loss 692.67\n",
            "2019-11-26 11:55:38,522 EPOCH 19\n",
            "2019-11-26 11:55:45,111 Epoch  19 Step:     7900 Batch Loss:     1.137055 Tokens per Sec:    14630, Lr: 0.000300\n",
            "2019-11-26 11:56:01,315 Epoch  19 Step:     8000 Batch Loss:     0.744475 Tokens per Sec:    15410, Lr: 0.000300\n",
            "2019-11-26 11:56:50,686 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 11:56:50,686 Saving new checkpoint.\n",
            "2019-11-26 11:56:50,965 Example #0\n",
            "2019-11-26 11:56:50,965 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 11:56:50,965 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:56:50,965 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 11:56:50,965 Example #1\n",
            "2019-11-26 11:56:50,966 \tSource:     The Drostdy\n",
            "2019-11-26 11:56:50,966 \tReference:  Die Drostdy\n",
            "2019-11-26 11:56:50,966 \tHypothesis: Die Drostdy\n",
            "2019-11-26 11:56:50,966 Example #2\n",
            "2019-11-26 11:56:50,966 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 11:56:50,966 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 11:56:50,966 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitoefening van persone buite die kommissie en om gemeenskapsbetrokkenheid te verseker dat die aktiwiteite van die kommissie .\n",
            "2019-11-26 11:56:50,966 Example #3\n",
            "2019-11-26 11:56:50,966 \tSource:     * private acts by individuals\n",
            "2019-11-26 11:56:50,966 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 11:56:50,966 \tHypothesis: * private aktiwiteite\n",
            "2019-11-26 11:56:50,967 Validation result (greedy) at epoch  19, step     8000: bleu:  37.38, loss: 37101.5312, ppl:   4.9042, duration: 49.6516s\n",
            "2019-11-26 11:57:07,168 Epoch  19 Step:     8100 Batch Loss:     1.343613 Tokens per Sec:    15250, Lr: 0.000300\n",
            "2019-11-26 11:57:23,183 Epoch  19 Step:     8200 Batch Loss:     1.560163 Tokens per Sec:    15112, Lr: 0.000300\n",
            "2019-11-26 11:57:38,611 Epoch  19: total training loss 668.08\n",
            "2019-11-26 11:57:38,611 EPOCH 20\n",
            "2019-11-26 11:57:39,619 Epoch  20 Step:     8300 Batch Loss:     1.303026 Tokens per Sec:    14111, Lr: 0.000300\n",
            "2019-11-26 11:57:55,953 Epoch  20 Step:     8400 Batch Loss:     2.189736 Tokens per Sec:    15124, Lr: 0.000300\n",
            "2019-11-26 11:58:12,254 Epoch  20 Step:     8500 Batch Loss:     2.371931 Tokens per Sec:    15080, Lr: 0.000300\n",
            "2019-11-26 11:58:28,533 Epoch  20 Step:     8600 Batch Loss:     1.199755 Tokens per Sec:    15320, Lr: 0.000300\n",
            "2019-11-26 11:58:44,679 Epoch  20 Step:     8700 Batch Loss:     1.696189 Tokens per Sec:    15200, Lr: 0.000300\n",
            "2019-11-26 11:58:49,286 Epoch  20: total training loss 658.15\n",
            "2019-11-26 11:58:49,286 EPOCH 21\n",
            "2019-11-26 11:59:01,062 Epoch  21 Step:     8800 Batch Loss:     1.814345 Tokens per Sec:    15202, Lr: 0.000300\n",
            "2019-11-26 11:59:17,383 Epoch  21 Step:     8900 Batch Loss:     1.208518 Tokens per Sec:    15373, Lr: 0.000300\n",
            "2019-11-26 11:59:33,675 Epoch  21 Step:     9000 Batch Loss:     1.584646 Tokens per Sec:    15159, Lr: 0.000300\n",
            "2019-11-26 11:59:49,908 Epoch  21 Step:     9100 Batch Loss:     1.363551 Tokens per Sec:    15324, Lr: 0.000300\n",
            "2019-11-26 11:59:59,784 Epoch  21: total training loss 634.78\n",
            "2019-11-26 11:59:59,785 EPOCH 22\n",
            "2019-11-26 12:00:05,863 Epoch  22 Step:     9200 Batch Loss:     1.234421 Tokens per Sec:    14720, Lr: 0.000300\n",
            "2019-11-26 12:00:22,161 Epoch  22 Step:     9300 Batch Loss:     1.366633 Tokens per Sec:    15356, Lr: 0.000300\n",
            "2019-11-26 12:00:38,320 Epoch  22 Step:     9400 Batch Loss:     0.922922 Tokens per Sec:    15111, Lr: 0.000300\n",
            "2019-11-26 12:00:54,446 Epoch  22 Step:     9500 Batch Loss:     1.129673 Tokens per Sec:    15157, Lr: 0.000300\n",
            "2019-11-26 12:01:10,329 Epoch  22: total training loss 620.19\n",
            "2019-11-26 12:01:10,330 EPOCH 23\n",
            "2019-11-26 12:01:10,872 Epoch  23 Step:     9600 Batch Loss:     0.834196 Tokens per Sec:    11898, Lr: 0.000300\n",
            "2019-11-26 12:01:27,012 Epoch  23 Step:     9700 Batch Loss:     1.974342 Tokens per Sec:    14733, Lr: 0.000300\n",
            "2019-11-26 12:01:43,280 Epoch  23 Step:     9800 Batch Loss:     1.712468 Tokens per Sec:    15026, Lr: 0.000300\n",
            "2019-11-26 12:01:59,561 Epoch  23 Step:     9900 Batch Loss:     1.192902 Tokens per Sec:    15332, Lr: 0.000300\n",
            "2019-11-26 12:02:15,717 Epoch  23 Step:    10000 Batch Loss:     1.113884 Tokens per Sec:    15262, Lr: 0.000300\n",
            "2019-11-26 12:03:05,106 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:03:05,107 Saving new checkpoint.\n",
            "2019-11-26 12:03:05,427 Example #0\n",
            "2019-11-26 12:03:05,428 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:03:05,428 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:03:05,428 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:03:05,428 Example #1\n",
            "2019-11-26 12:03:05,428 \tSource:     The Drostdy\n",
            "2019-11-26 12:03:05,428 \tReference:  Die Drostdy\n",
            "2019-11-26 12:03:05,428 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:03:05,428 Example #2\n",
            "2019-11-26 12:03:05,428 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:03:05,428 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:03:05,429 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitoefening van persone buite die kommissie en om gemeenskapsbetrokkenheid te verseker in die aktiwiteite van die kommissie .\n",
            "2019-11-26 12:03:05,429 Example #3\n",
            "2019-11-26 12:03:05,429 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:03:05,429 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:03:05,429 \tHypothesis: * private aktiwiteite\n",
            "2019-11-26 12:03:05,429 Validation result (greedy) at epoch  23, step    10000: bleu:  39.24, loss: 34156.6953, ppl:   4.3227, duration: 49.7111s\n",
            "2019-11-26 12:03:11,229 Epoch  23: total training loss 615.70\n",
            "2019-11-26 12:03:11,229 EPOCH 24\n",
            "2019-11-26 12:03:21,789 Epoch  24 Step:    10100 Batch Loss:     1.073194 Tokens per Sec:    15289, Lr: 0.000300\n",
            "2019-11-26 12:03:37,939 Epoch  24 Step:    10200 Batch Loss:     0.900523 Tokens per Sec:    15149, Lr: 0.000300\n",
            "2019-11-26 12:03:53,980 Epoch  24 Step:    10300 Batch Loss:     0.963902 Tokens per Sec:    14954, Lr: 0.000300\n",
            "2019-11-26 12:04:09,997 Epoch  24 Step:    10400 Batch Loss:     2.269618 Tokens per Sec:    15070, Lr: 0.000300\n",
            "2019-11-26 12:04:21,952 Epoch  24: total training loss 601.49\n",
            "2019-11-26 12:04:21,953 EPOCH 25\n",
            "2019-11-26 12:04:26,251 Epoch  25 Step:    10500 Batch Loss:     0.856459 Tokens per Sec:    15093, Lr: 0.000300\n",
            "2019-11-26 12:04:42,325 Epoch  25 Step:    10600 Batch Loss:     1.371791 Tokens per Sec:    14951, Lr: 0.000300\n",
            "2019-11-26 12:04:58,329 Epoch  25 Step:    10700 Batch Loss:     2.013452 Tokens per Sec:    15087, Lr: 0.000300\n",
            "2019-11-26 12:05:14,811 Epoch  25 Step:    10800 Batch Loss:     1.043119 Tokens per Sec:    15628, Lr: 0.000300\n",
            "2019-11-26 12:05:30,824 Epoch  25 Step:    10900 Batch Loss:     1.488485 Tokens per Sec:    15095, Lr: 0.000300\n",
            "2019-11-26 12:05:32,691 Epoch  25: total training loss 585.86\n",
            "2019-11-26 12:05:32,691 EPOCH 26\n",
            "2019-11-26 12:05:46,973 Epoch  26 Step:    11000 Batch Loss:     1.354562 Tokens per Sec:    14988, Lr: 0.000300\n",
            "2019-11-26 12:06:03,166 Epoch  26 Step:    11100 Batch Loss:     1.148171 Tokens per Sec:    15192, Lr: 0.000300\n",
            "2019-11-26 12:06:19,367 Epoch  26 Step:    11200 Batch Loss:     1.528958 Tokens per Sec:    15187, Lr: 0.000300\n",
            "2019-11-26 12:06:35,624 Epoch  26 Step:    11300 Batch Loss:     1.651470 Tokens per Sec:    15094, Lr: 0.000300\n",
            "2019-11-26 12:06:43,633 Epoch  26: total training loss 572.71\n",
            "2019-11-26 12:06:43,634 EPOCH 27\n",
            "2019-11-26 12:06:51,731 Epoch  27 Step:    11400 Batch Loss:     0.593972 Tokens per Sec:    14497, Lr: 0.000300\n",
            "2019-11-26 12:07:07,731 Epoch  27 Step:    11500 Batch Loss:     1.662266 Tokens per Sec:    15226, Lr: 0.000300\n",
            "2019-11-26 12:07:24,086 Epoch  27 Step:    11600 Batch Loss:     1.297733 Tokens per Sec:    15404, Lr: 0.000300\n",
            "2019-11-26 12:07:40,217 Epoch  27 Step:    11700 Batch Loss:     2.105778 Tokens per Sec:    15112, Lr: 0.000300\n",
            "2019-11-26 12:07:54,329 Epoch  27: total training loss 564.17\n",
            "2019-11-26 12:07:54,329 EPOCH 28\n",
            "2019-11-26 12:07:56,491 Epoch  28 Step:    11800 Batch Loss:     0.865440 Tokens per Sec:    14615, Lr: 0.000300\n",
            "2019-11-26 12:08:12,484 Epoch  28 Step:    11900 Batch Loss:     1.723002 Tokens per Sec:    15050, Lr: 0.000300\n",
            "2019-11-26 12:08:28,696 Epoch  28 Step:    12000 Batch Loss:     1.384915 Tokens per Sec:    15254, Lr: 0.000300\n",
            "2019-11-26 12:09:17,865 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:09:17,866 Saving new checkpoint.\n",
            "2019-11-26 12:09:18,163 Example #0\n",
            "2019-11-26 12:09:18,164 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:09:18,164 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:09:18,164 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:09:18,164 Example #1\n",
            "2019-11-26 12:09:18,165 \tSource:     The Drostdy\n",
            "2019-11-26 12:09:18,165 \tReference:  Die Drostdy\n",
            "2019-11-26 12:09:18,165 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:09:18,165 Example #2\n",
            "2019-11-26 12:09:18,165 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:09:18,165 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:09:18,165 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitoefening van persone buite die kommissie en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:09:18,165 Example #3\n",
            "2019-11-26 12:09:18,166 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:09:18,166 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:09:18,166 \tHypothesis: * private aktiwitte deur individue\n",
            "2019-11-26 12:09:18,166 Validation result (greedy) at epoch  28, step    12000: bleu:  40.75, loss: 32103.4766, ppl:   3.9586, duration: 49.4692s\n",
            "2019-11-26 12:09:34,605 Epoch  28 Step:    12100 Batch Loss:     0.898625 Tokens per Sec:    15389, Lr: 0.000300\n",
            "2019-11-26 12:09:50,813 Epoch  28 Step:    12200 Batch Loss:     0.866637 Tokens per Sec:    15133, Lr: 0.000300\n",
            "2019-11-26 12:09:54,228 Epoch  28: total training loss 545.53\n",
            "2019-11-26 12:09:54,228 EPOCH 29\n",
            "2019-11-26 12:10:07,395 Epoch  29 Step:    12300 Batch Loss:     0.867669 Tokens per Sec:    15033, Lr: 0.000300\n",
            "2019-11-26 12:10:23,427 Epoch  29 Step:    12400 Batch Loss:     1.148031 Tokens per Sec:    15062, Lr: 0.000300\n",
            "2019-11-26 12:10:39,675 Epoch  29 Step:    12500 Batch Loss:     0.895892 Tokens per Sec:    15196, Lr: 0.000300\n",
            "2019-11-26 12:10:56,010 Epoch  29 Step:    12600 Batch Loss:     1.510698 Tokens per Sec:    15395, Lr: 0.000300\n",
            "2019-11-26 12:11:04,947 Epoch  29: total training loss 541.54\n",
            "2019-11-26 12:11:04,947 EPOCH 30\n",
            "2019-11-26 12:11:12,165 Epoch  30 Step:    12700 Batch Loss:     0.837670 Tokens per Sec:    15142, Lr: 0.000300\n",
            "2019-11-26 12:11:28,433 Epoch  30 Step:    12800 Batch Loss:     1.230110 Tokens per Sec:    15259, Lr: 0.000300\n",
            "2019-11-26 12:11:44,668 Epoch  30 Step:    12900 Batch Loss:     1.488923 Tokens per Sec:    15090, Lr: 0.000300\n",
            "2019-11-26 12:12:00,827 Epoch  30 Step:    13000 Batch Loss:     1.608192 Tokens per Sec:    15088, Lr: 0.000300\n",
            "2019-11-26 12:12:15,873 Epoch  30: total training loss 536.01\n",
            "2019-11-26 12:12:15,873 EPOCH 31\n",
            "2019-11-26 12:12:17,066 Epoch  31 Step:    13100 Batch Loss:     0.685046 Tokens per Sec:    13524, Lr: 0.000300\n",
            "2019-11-26 12:12:33,255 Epoch  31 Step:    13200 Batch Loss:     1.043526 Tokens per Sec:    15304, Lr: 0.000300\n",
            "2019-11-26 12:12:49,554 Epoch  31 Step:    13300 Batch Loss:     1.331799 Tokens per Sec:    15257, Lr: 0.000300\n",
            "2019-11-26 12:13:05,761 Epoch  31 Step:    13400 Batch Loss:     0.562917 Tokens per Sec:    15260, Lr: 0.000300\n",
            "2019-11-26 12:13:21,805 Epoch  31 Step:    13500 Batch Loss:     0.871661 Tokens per Sec:    15070, Lr: 0.000300\n",
            "2019-11-26 12:13:26,400 Epoch  31: total training loss 522.61\n",
            "2019-11-26 12:13:26,400 EPOCH 32\n",
            "2019-11-26 12:13:38,142 Epoch  32 Step:    13600 Batch Loss:     1.529986 Tokens per Sec:    15354, Lr: 0.000300\n",
            "2019-11-26 12:13:54,424 Epoch  32 Step:    13700 Batch Loss:     0.749274 Tokens per Sec:    15151, Lr: 0.000300\n",
            "2019-11-26 12:14:10,606 Epoch  32 Step:    13800 Batch Loss:     1.366758 Tokens per Sec:    15268, Lr: 0.000300\n",
            "2019-11-26 12:14:26,717 Epoch  32 Step:    13900 Batch Loss:     0.794766 Tokens per Sec:    15063, Lr: 0.000300\n",
            "2019-11-26 12:14:36,969 Epoch  32: total training loss 515.14\n",
            "2019-11-26 12:14:36,970 EPOCH 33\n",
            "2019-11-26 12:14:43,040 Epoch  33 Step:    14000 Batch Loss:     1.347236 Tokens per Sec:    15130, Lr: 0.000300\n",
            "2019-11-26 12:15:32,235 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:15:32,235 Saving new checkpoint.\n",
            "2019-11-26 12:15:32,529 Example #0\n",
            "2019-11-26 12:15:32,530 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:15:32,530 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:15:32,530 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:15:32,530 Example #1\n",
            "2019-11-26 12:15:32,530 \tSource:     The Drostdy\n",
            "2019-11-26 12:15:32,530 \tReference:  Die Drostdy\n",
            "2019-11-26 12:15:32,530 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:15:32,531 Example #2\n",
            "2019-11-26 12:15:32,531 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:15:32,531 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:15:32,531 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitoefening van persone buite die kommissie te verrig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:15:32,531 Example #3\n",
            "2019-11-26 12:15:32,531 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:15:32,531 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:15:32,531 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 12:15:32,531 Validation result (greedy) at epoch  33, step    14000: bleu:  42.70, loss: 30750.4551, ppl:   3.7356, duration: 49.4913s\n",
            "2019-11-26 12:15:48,801 Epoch  33 Step:    14100 Batch Loss:     1.323262 Tokens per Sec:    15296, Lr: 0.000300\n",
            "2019-11-26 12:16:04,902 Epoch  33 Step:    14200 Batch Loss:     0.639752 Tokens per Sec:    15164, Lr: 0.000300\n",
            "2019-11-26 12:16:20,847 Epoch  33 Step:    14300 Batch Loss:     1.079608 Tokens per Sec:    15209, Lr: 0.000300\n",
            "2019-11-26 12:16:37,000 Epoch  33 Step:    14400 Batch Loss:     1.483253 Tokens per Sec:    15126, Lr: 0.000300\n",
            "2019-11-26 12:16:37,002 Epoch  33: total training loss 507.69\n",
            "2019-11-26 12:16:37,002 EPOCH 34\n",
            "2019-11-26 12:16:53,186 Epoch  34 Step:    14500 Batch Loss:     1.170611 Tokens per Sec:    14936, Lr: 0.000300\n",
            "2019-11-26 12:17:09,308 Epoch  34 Step:    14600 Batch Loss:     0.669251 Tokens per Sec:    15056, Lr: 0.000300\n",
            "2019-11-26 12:17:25,362 Epoch  34 Step:    14700 Batch Loss:     1.111993 Tokens per Sec:    15259, Lr: 0.000300\n",
            "2019-11-26 12:17:41,577 Epoch  34 Step:    14800 Batch Loss:     1.475204 Tokens per Sec:    15252, Lr: 0.000300\n",
            "2019-11-26 12:17:47,862 Epoch  34: total training loss 503.21\n",
            "2019-11-26 12:17:47,863 EPOCH 35\n",
            "2019-11-26 12:17:57,760 Epoch  35 Step:    14900 Batch Loss:     0.921647 Tokens per Sec:    15098, Lr: 0.000300\n",
            "2019-11-26 12:18:13,861 Epoch  35 Step:    15000 Batch Loss:     1.141478 Tokens per Sec:    15162, Lr: 0.000300\n",
            "2019-11-26 12:18:30,162 Epoch  35 Step:    15100 Batch Loss:     0.792189 Tokens per Sec:    15325, Lr: 0.000300\n",
            "2019-11-26 12:18:46,250 Epoch  35 Step:    15200 Batch Loss:     0.894321 Tokens per Sec:    15142, Lr: 0.000300\n",
            "2019-11-26 12:18:58,375 Epoch  35: total training loss 491.17\n",
            "2019-11-26 12:18:58,375 EPOCH 36\n",
            "2019-11-26 12:19:02,451 Epoch  36 Step:    15300 Batch Loss:     1.113939 Tokens per Sec:    15135, Lr: 0.000300\n",
            "2019-11-26 12:19:18,516 Epoch  36 Step:    15400 Batch Loss:     0.942997 Tokens per Sec:    15060, Lr: 0.000300\n",
            "2019-11-26 12:19:34,748 Epoch  36 Step:    15500 Batch Loss:     1.089445 Tokens per Sec:    15092, Lr: 0.000300\n",
            "2019-11-26 12:19:50,960 Epoch  36 Step:    15600 Batch Loss:     0.848535 Tokens per Sec:    15393, Lr: 0.000300\n",
            "2019-11-26 12:20:07,044 Epoch  36 Step:    15700 Batch Loss:     0.793746 Tokens per Sec:    15296, Lr: 0.000300\n",
            "2019-11-26 12:20:08,866 Epoch  36: total training loss 485.58\n",
            "2019-11-26 12:20:08,866 EPOCH 37\n",
            "2019-11-26 12:20:23,278 Epoch  37 Step:    15800 Batch Loss:     1.291795 Tokens per Sec:    15147, Lr: 0.000300\n",
            "2019-11-26 12:20:39,626 Epoch  37 Step:    15900 Batch Loss:     0.503743 Tokens per Sec:    15214, Lr: 0.000300\n",
            "2019-11-26 12:20:55,779 Epoch  37 Step:    16000 Batch Loss:     1.535897 Tokens per Sec:    15183, Lr: 0.000300\n",
            "2019-11-26 12:21:44,969 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:21:44,969 Saving new checkpoint.\n",
            "2019-11-26 12:21:45,283 Example #0\n",
            "2019-11-26 12:21:45,283 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:21:45,283 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:21:45,283 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:21:45,284 Example #1\n",
            "2019-11-26 12:21:45,284 \tSource:     The Drostdy\n",
            "2019-11-26 12:21:45,284 \tReference:  Die Drostdy\n",
            "2019-11-26 12:21:45,284 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:21:45,284 Example #2\n",
            "2019-11-26 12:21:45,284 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:21:45,284 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:21:45,285 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoering van persone buite die kommissie en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:21:45,285 Example #3\n",
            "2019-11-26 12:21:45,285 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:21:45,285 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:21:45,285 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 12:21:45,286 Validation result (greedy) at epoch  37, step    16000: bleu:  43.53, loss: 29685.3848, ppl:   3.5689, duration: 49.5060s\n",
            "2019-11-26 12:22:01,514 Epoch  37 Step:    16100 Batch Loss:     0.977506 Tokens per Sec:    15133, Lr: 0.000300\n",
            "2019-11-26 12:22:09,019 Epoch  37: total training loss 477.42\n",
            "2019-11-26 12:22:09,019 EPOCH 38\n",
            "2019-11-26 12:22:17,862 Epoch  38 Step:    16200 Batch Loss:     0.912274 Tokens per Sec:    15267, Lr: 0.000300\n",
            "2019-11-26 12:22:33,879 Epoch  38 Step:    16300 Batch Loss:     0.727981 Tokens per Sec:    15001, Lr: 0.000300\n",
            "2019-11-26 12:22:49,879 Epoch  38 Step:    16400 Batch Loss:     1.225804 Tokens per Sec:    15153, Lr: 0.000300\n",
            "2019-11-26 12:23:06,183 Epoch  38 Step:    16500 Batch Loss:     1.238150 Tokens per Sec:    15327, Lr: 0.000300\n",
            "2019-11-26 12:23:19,612 Epoch  38: total training loss 473.37\n",
            "2019-11-26 12:23:19,612 EPOCH 39\n",
            "2019-11-26 12:23:22,509 Epoch  39 Step:    16600 Batch Loss:     1.538846 Tokens per Sec:    15618, Lr: 0.000300\n",
            "2019-11-26 12:23:38,561 Epoch  39 Step:    16700 Batch Loss:     1.262424 Tokens per Sec:    15037, Lr: 0.000300\n",
            "2019-11-26 12:23:54,833 Epoch  39 Step:    16800 Batch Loss:     0.833469 Tokens per Sec:    15480, Lr: 0.000300\n",
            "2019-11-26 12:24:10,897 Epoch  39 Step:    16900 Batch Loss:     1.285361 Tokens per Sec:    15151, Lr: 0.000300\n",
            "2019-11-26 12:24:27,113 Epoch  39 Step:    17000 Batch Loss:     1.659389 Tokens per Sec:    15363, Lr: 0.000300\n",
            "2019-11-26 12:24:29,950 Epoch  39: total training loss 466.35\n",
            "2019-11-26 12:24:29,950 EPOCH 40\n",
            "2019-11-26 12:24:43,272 Epoch  40 Step:    17100 Batch Loss:     1.011768 Tokens per Sec:    15085, Lr: 0.000300\n",
            "2019-11-26 12:24:59,561 Epoch  40 Step:    17200 Batch Loss:     0.899179 Tokens per Sec:    15377, Lr: 0.000300\n",
            "2019-11-26 12:25:15,546 Epoch  40 Step:    17300 Batch Loss:     1.233902 Tokens per Sec:    15265, Lr: 0.000300\n",
            "2019-11-26 12:25:31,671 Epoch  40 Step:    17400 Batch Loss:     0.851410 Tokens per Sec:    15242, Lr: 0.000300\n",
            "2019-11-26 12:25:40,139 Epoch  40: total training loss 457.48\n",
            "2019-11-26 12:25:40,139 EPOCH 41\n",
            "2019-11-26 12:25:47,817 Epoch  41 Step:    17500 Batch Loss:     0.938755 Tokens per Sec:    15240, Lr: 0.000300\n",
            "2019-11-26 12:26:04,211 Epoch  41 Step:    17600 Batch Loss:     0.670900 Tokens per Sec:    15533, Lr: 0.000300\n",
            "2019-11-26 12:26:20,520 Epoch  41 Step:    17700 Batch Loss:     1.540982 Tokens per Sec:    15310, Lr: 0.000300\n",
            "2019-11-26 12:26:36,599 Epoch  41 Step:    17800 Batch Loss:     0.465548 Tokens per Sec:    15153, Lr: 0.000300\n",
            "2019-11-26 12:26:50,603 Epoch  41: total training loss 456.14\n",
            "2019-11-26 12:26:50,603 EPOCH 42\n",
            "2019-11-26 12:26:52,389 Epoch  42 Step:    17900 Batch Loss:     1.453590 Tokens per Sec:    14513, Lr: 0.000300\n",
            "2019-11-26 12:27:08,452 Epoch  42 Step:    18000 Batch Loss:     0.585894 Tokens per Sec:    15103, Lr: 0.000300\n",
            "2019-11-26 12:27:57,678 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:27:57,678 Saving new checkpoint.\n",
            "2019-11-26 12:27:57,986 Example #0\n",
            "2019-11-26 12:27:57,987 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:27:57,987 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:27:57,987 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:27:57,987 Example #1\n",
            "2019-11-26 12:27:57,987 \tSource:     The Drostdy\n",
            "2019-11-26 12:27:57,987 \tReference:  Die Drostdy\n",
            "2019-11-26 12:27:57,987 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:27:57,987 Example #2\n",
            "2019-11-26 12:27:57,987 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:27:57,987 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:27:57,988 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die prestasie van mense buite die kommissie te verrig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:27:57,988 Example #3\n",
            "2019-11-26 12:27:57,988 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:27:57,988 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:27:57,988 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 12:27:57,988 Validation result (greedy) at epoch  42, step    18000: bleu:  44.48, loss: 28864.5508, ppl:   3.4455, duration: 49.5352s\n",
            "2019-11-26 12:28:14,099 Epoch  42 Step:    18100 Batch Loss:     1.758319 Tokens per Sec:    15362, Lr: 0.000300\n",
            "2019-11-26 12:28:30,021 Epoch  42 Step:    18200 Batch Loss:     1.352633 Tokens per Sec:    15264, Lr: 0.000300\n",
            "2019-11-26 12:28:46,132 Epoch  42 Step:    18300 Batch Loss:     0.403492 Tokens per Sec:    15227, Lr: 0.000300\n",
            "2019-11-26 12:28:50,681 Epoch  42: total training loss 454.43\n",
            "2019-11-26 12:28:50,681 EPOCH 43\n",
            "2019-11-26 12:29:02,410 Epoch  43 Step:    18400 Batch Loss:     1.042653 Tokens per Sec:    15309, Lr: 0.000300\n",
            "2019-11-26 12:29:18,434 Epoch  43 Step:    18500 Batch Loss:     1.022686 Tokens per Sec:    15217, Lr: 0.000300\n",
            "2019-11-26 12:29:34,598 Epoch  43 Step:    18600 Batch Loss:     1.028972 Tokens per Sec:    15289, Lr: 0.000300\n",
            "2019-11-26 12:29:50,739 Epoch  43 Step:    18700 Batch Loss:     0.827851 Tokens per Sec:    15141, Lr: 0.000300\n",
            "2019-11-26 12:30:01,134 Epoch  43: total training loss 446.05\n",
            "2019-11-26 12:30:01,134 EPOCH 44\n",
            "2019-11-26 12:30:06,654 Epoch  44 Step:    18800 Batch Loss:     1.084404 Tokens per Sec:    15116, Lr: 0.000300\n",
            "2019-11-26 12:30:22,742 Epoch  44 Step:    18900 Batch Loss:     1.109812 Tokens per Sec:    15308, Lr: 0.000300\n",
            "2019-11-26 12:30:38,770 Epoch  44 Step:    19000 Batch Loss:     1.085626 Tokens per Sec:    15304, Lr: 0.000300\n",
            "2019-11-26 12:30:54,953 Epoch  44 Step:    19100 Batch Loss:     1.026974 Tokens per Sec:    15263, Lr: 0.000300\n",
            "2019-11-26 12:31:10,750 Epoch  44 Step:    19200 Batch Loss:     1.237570 Tokens per Sec:    14984, Lr: 0.000300\n",
            "2019-11-26 12:31:11,552 Epoch  44: total training loss 440.61\n",
            "2019-11-26 12:31:11,553 EPOCH 45\n",
            "2019-11-26 12:31:26,641 Epoch  45 Step:    19300 Batch Loss:     1.067398 Tokens per Sec:    14833, Lr: 0.000300\n",
            "2019-11-26 12:31:42,861 Epoch  45 Step:    19400 Batch Loss:     1.251102 Tokens per Sec:    15620, Lr: 0.000300\n",
            "2019-11-26 12:31:58,694 Epoch  45 Step:    19500 Batch Loss:     1.193938 Tokens per Sec:    15227, Lr: 0.000300\n",
            "2019-11-26 12:32:14,845 Epoch  45 Step:    19600 Batch Loss:     0.696879 Tokens per Sec:    15173, Lr: 0.000300\n",
            "2019-11-26 12:32:21,965 Epoch  45: total training loss 436.23\n",
            "2019-11-26 12:32:21,966 EPOCH 46\n",
            "2019-11-26 12:32:31,142 Epoch  46 Step:    19700 Batch Loss:     1.393640 Tokens per Sec:    15139, Lr: 0.000300\n",
            "2019-11-26 12:32:47,113 Epoch  46 Step:    19800 Batch Loss:     0.759225 Tokens per Sec:    15201, Lr: 0.000300\n",
            "2019-11-26 12:33:03,042 Epoch  46 Step:    19900 Batch Loss:     1.119920 Tokens per Sec:    15071, Lr: 0.000300\n",
            "2019-11-26 12:33:19,344 Epoch  46 Step:    20000 Batch Loss:     1.032209 Tokens per Sec:    15495, Lr: 0.000300\n",
            "2019-11-26 12:34:08,487 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:34:08,488 Saving new checkpoint.\n",
            "2019-11-26 12:34:08,793 Example #0\n",
            "2019-11-26 12:34:08,793 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:34:08,793 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:34:08,793 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:34:08,793 Example #1\n",
            "2019-11-26 12:34:08,793 \tSource:     The Drostdy\n",
            "2019-11-26 12:34:08,793 \tReference:  Die Drostdy\n",
            "2019-11-26 12:34:08,794 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:34:08,794 Example #2\n",
            "2019-11-26 12:34:08,794 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:34:08,794 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:34:08,794 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die prestasie van persone buite die kommissie te verrig en om gemeenskapsbetrokkenheid te verseker in die aktiwiteite van die kommissie .\n",
            "2019-11-26 12:34:08,794 Example #3\n",
            "2019-11-26 12:34:08,794 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:34:08,794 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:34:08,794 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 12:34:08,794 Validation result (greedy) at epoch  46, step    20000: bleu:  45.29, loss: 28165.5215, ppl:   3.3438, duration: 49.4499s\n",
            "2019-11-26 12:34:21,853 Epoch  46: total training loss 433.22\n",
            "2019-11-26 12:34:21,853 EPOCH 47\n",
            "2019-11-26 12:34:24,766 Epoch  47 Step:    20100 Batch Loss:     0.468254 Tokens per Sec:    14227, Lr: 0.000300\n",
            "2019-11-26 12:34:40,873 Epoch  47 Step:    20200 Batch Loss:     1.215164 Tokens per Sec:    15341, Lr: 0.000300\n",
            "2019-11-26 12:34:56,979 Epoch  47 Step:    20300 Batch Loss:     0.461400 Tokens per Sec:    15495, Lr: 0.000300\n",
            "2019-11-26 12:35:13,060 Epoch  47 Step:    20400 Batch Loss:     0.578643 Tokens per Sec:    15084, Lr: 0.000300\n",
            "2019-11-26 12:35:29,084 Epoch  47 Step:    20500 Batch Loss:     1.248637 Tokens per Sec:    15181, Lr: 0.000300\n",
            "2019-11-26 12:35:32,298 Epoch  47: total training loss 425.75\n",
            "2019-11-26 12:35:32,298 EPOCH 48\n",
            "2019-11-26 12:35:45,305 Epoch  48 Step:    20600 Batch Loss:     0.957772 Tokens per Sec:    15184, Lr: 0.000300\n",
            "2019-11-26 12:36:01,312 Epoch  48 Step:    20700 Batch Loss:     1.181914 Tokens per Sec:    15048, Lr: 0.000300\n",
            "2019-11-26 12:36:17,330 Epoch  48 Step:    20800 Batch Loss:     0.418090 Tokens per Sec:    15133, Lr: 0.000300\n",
            "2019-11-26 12:36:33,632 Epoch  48 Step:    20900 Batch Loss:     1.357821 Tokens per Sec:    15282, Lr: 0.000300\n",
            "2019-11-26 12:36:42,844 Epoch  48: total training loss 421.14\n",
            "2019-11-26 12:36:42,844 EPOCH 49\n",
            "2019-11-26 12:36:49,969 Epoch  49 Step:    21000 Batch Loss:     1.088681 Tokens per Sec:    15064, Lr: 0.000300\n",
            "2019-11-26 12:37:05,778 Epoch  49 Step:    21100 Batch Loss:     0.738177 Tokens per Sec:    15012, Lr: 0.000300\n",
            "2019-11-26 12:37:21,886 Epoch  49 Step:    21200 Batch Loss:     0.707294 Tokens per Sec:    15294, Lr: 0.000300\n",
            "2019-11-26 12:37:38,156 Epoch  49 Step:    21300 Batch Loss:     1.099869 Tokens per Sec:    15548, Lr: 0.000300\n",
            "2019-11-26 12:37:53,316 Epoch  49: total training loss 424.99\n",
            "2019-11-26 12:37:53,316 EPOCH 50\n",
            "2019-11-26 12:37:54,338 Epoch  50 Step:    21400 Batch Loss:     1.343167 Tokens per Sec:    13686, Lr: 0.000300\n",
            "2019-11-26 12:38:10,322 Epoch  50 Step:    21500 Batch Loss:     1.236026 Tokens per Sec:    15273, Lr: 0.000300\n",
            "2019-11-26 12:38:26,352 Epoch  50 Step:    21600 Batch Loss:     0.770427 Tokens per Sec:    15153, Lr: 0.000300\n",
            "2019-11-26 12:38:42,469 Epoch  50 Step:    21700 Batch Loss:     0.912537 Tokens per Sec:    15479, Lr: 0.000300\n",
            "2019-11-26 12:38:58,545 Epoch  50 Step:    21800 Batch Loss:     0.712236 Tokens per Sec:    15273, Lr: 0.000300\n",
            "2019-11-26 12:39:03,509 Epoch  50: total training loss 415.63\n",
            "2019-11-26 12:39:03,510 EPOCH 51\n",
            "2019-11-26 12:39:14,675 Epoch  51 Step:    21900 Batch Loss:     0.610291 Tokens per Sec:    15019, Lr: 0.000300\n",
            "2019-11-26 12:39:30,690 Epoch  51 Step:    22000 Batch Loss:     1.360288 Tokens per Sec:    15119, Lr: 0.000300\n",
            "2019-11-26 12:40:19,739 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:40:19,740 Saving new checkpoint.\n",
            "2019-11-26 12:40:20,033 Example #0\n",
            "2019-11-26 12:40:20,034 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:40:20,034 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:40:20,034 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:40:20,034 Example #1\n",
            "2019-11-26 12:40:20,034 \tSource:     The Drostdy\n",
            "2019-11-26 12:40:20,034 \tReference:  Die Drostdy\n",
            "2019-11-26 12:40:20,034 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:40:20,034 Example #2\n",
            "2019-11-26 12:40:20,034 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:40:20,034 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:40:20,034 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoer van persone buite die kommissie te verrig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:40:20,034 Example #3\n",
            "2019-11-26 12:40:20,034 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:40:20,035 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:40:20,035 \tHypothesis: * private aktiwiteite deur individue\n",
            "2019-11-26 12:40:20,035 Validation result (greedy) at epoch  51, step    22000: bleu:  45.67, loss: 27861.9648, ppl:   3.3006, duration: 49.3443s\n",
            "2019-11-26 12:40:35,990 Epoch  51 Step:    22100 Batch Loss:     0.817147 Tokens per Sec:    15144, Lr: 0.000300\n",
            "2019-11-26 12:40:52,247 Epoch  51 Step:    22200 Batch Loss:     0.663797 Tokens per Sec:    15307, Lr: 0.000300\n",
            "2019-11-26 12:41:03,320 Epoch  51: total training loss 408.61\n",
            "2019-11-26 12:41:03,320 EPOCH 52\n",
            "2019-11-26 12:41:08,379 Epoch  52 Step:    22300 Batch Loss:     0.925180 Tokens per Sec:    15338, Lr: 0.000300\n",
            "2019-11-26 12:41:24,295 Epoch  52 Step:    22400 Batch Loss:     1.305108 Tokens per Sec:    15390, Lr: 0.000300\n",
            "2019-11-26 12:41:40,358 Epoch  52 Step:    22500 Batch Loss:     0.610038 Tokens per Sec:    15234, Lr: 0.000300\n",
            "2019-11-26 12:41:56,407 Epoch  52 Step:    22600 Batch Loss:     0.951900 Tokens per Sec:    15227, Lr: 0.000300\n",
            "2019-11-26 12:42:12,500 Epoch  52 Step:    22700 Batch Loss:     1.455232 Tokens per Sec:    15229, Lr: 0.000300\n",
            "2019-11-26 12:42:13,615 Epoch  52: total training loss 408.93\n",
            "2019-11-26 12:42:13,616 EPOCH 53\n",
            "2019-11-26 12:42:28,584 Epoch  53 Step:    22800 Batch Loss:     0.811055 Tokens per Sec:    15221, Lr: 0.000300\n",
            "2019-11-26 12:42:44,659 Epoch  53 Step:    22900 Batch Loss:     0.992817 Tokens per Sec:    15176, Lr: 0.000300\n",
            "2019-11-26 12:43:00,892 Epoch  53 Step:    23000 Batch Loss:     1.011394 Tokens per Sec:    15145, Lr: 0.000300\n",
            "2019-11-26 12:43:16,895 Epoch  53 Step:    23100 Batch Loss:     0.627608 Tokens per Sec:    15258, Lr: 0.000300\n",
            "2019-11-26 12:43:23,899 Epoch  53: total training loss 399.86\n",
            "2019-11-26 12:43:23,900 EPOCH 54\n",
            "2019-11-26 12:43:33,224 Epoch  54 Step:    23200 Batch Loss:     0.776448 Tokens per Sec:    15413, Lr: 0.000300\n",
            "2019-11-26 12:43:49,284 Epoch  54 Step:    23300 Batch Loss:     0.541824 Tokens per Sec:    15024, Lr: 0.000300\n",
            "2019-11-26 12:44:05,413 Epoch  54 Step:    23400 Batch Loss:     1.097711 Tokens per Sec:    15589, Lr: 0.000300\n",
            "2019-11-26 12:44:21,358 Epoch  54 Step:    23500 Batch Loss:     0.652817 Tokens per Sec:    14880, Lr: 0.000300\n",
            "2019-11-26 12:44:34,162 Epoch  54: total training loss 395.18\n",
            "2019-11-26 12:44:34,162 EPOCH 55\n",
            "2019-11-26 12:44:37,631 Epoch  55 Step:    23600 Batch Loss:     0.653211 Tokens per Sec:    14899, Lr: 0.000300\n",
            "2019-11-26 12:44:53,779 Epoch  55 Step:    23700 Batch Loss:     0.817440 Tokens per Sec:    15327, Lr: 0.000300\n",
            "2019-11-26 12:45:09,858 Epoch  55 Step:    23800 Batch Loss:     1.058987 Tokens per Sec:    15085, Lr: 0.000300\n",
            "2019-11-26 12:45:26,080 Epoch  55 Step:    23900 Batch Loss:     1.008387 Tokens per Sec:    15448, Lr: 0.000300\n",
            "2019-11-26 12:45:42,389 Epoch  55 Step:    24000 Batch Loss:     0.760989 Tokens per Sec:    15439, Lr: 0.000300\n",
            "2019-11-26 12:46:31,554 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:46:31,554 Saving new checkpoint.\n",
            "2019-11-26 12:46:31,862 Example #0\n",
            "2019-11-26 12:46:31,862 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:46:31,862 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:46:31,862 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:46:31,862 Example #1\n",
            "2019-11-26 12:46:31,862 \tSource:     The Drostdy\n",
            "2019-11-26 12:46:31,862 \tReference:  Die Drostdy\n",
            "2019-11-26 12:46:31,862 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:46:31,862 Example #2\n",
            "2019-11-26 12:46:31,862 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:46:31,862 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:46:31,863 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoering van persone buite die kommissie te verrig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:46:31,863 Example #3\n",
            "2019-11-26 12:46:31,863 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:46:31,863 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:46:31,863 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 12:46:31,863 Validation result (greedy) at epoch  55, step    24000: bleu:  47.06, loss: 27393.8047, ppl:   3.2350, duration: 49.4739s\n",
            "2019-11-26 12:46:33,675 Epoch  55: total training loss 388.99\n",
            "2019-11-26 12:46:33,676 EPOCH 56\n",
            "2019-11-26 12:46:47,830 Epoch  56 Step:    24100 Batch Loss:     0.435184 Tokens per Sec:    14840, Lr: 0.000300\n",
            "2019-11-26 12:47:03,916 Epoch  56 Step:    24200 Batch Loss:     0.777254 Tokens per Sec:    15185, Lr: 0.000300\n",
            "2019-11-26 12:47:20,008 Epoch  56 Step:    24300 Batch Loss:     1.170903 Tokens per Sec:    15558, Lr: 0.000300\n",
            "2019-11-26 12:47:36,393 Epoch  56 Step:    24400 Batch Loss:     0.976754 Tokens per Sec:    15367, Lr: 0.000300\n",
            "2019-11-26 12:47:44,006 Epoch  56: total training loss 388.88\n",
            "2019-11-26 12:47:44,006 EPOCH 57\n",
            "2019-11-26 12:47:52,646 Epoch  57 Step:    24500 Batch Loss:     0.735243 Tokens per Sec:    15321, Lr: 0.000300\n",
            "2019-11-26 12:48:08,770 Epoch  57 Step:    24600 Batch Loss:     1.187664 Tokens per Sec:    15045, Lr: 0.000300\n",
            "2019-11-26 12:48:24,897 Epoch  57 Step:    24700 Batch Loss:     1.073763 Tokens per Sec:    15125, Lr: 0.000300\n",
            "2019-11-26 12:48:41,008 Epoch  57 Step:    24800 Batch Loss:     0.665125 Tokens per Sec:    15157, Lr: 0.000300\n",
            "2019-11-26 12:48:54,409 Epoch  57: total training loss 384.62\n",
            "2019-11-26 12:48:54,409 EPOCH 58\n",
            "2019-11-26 12:48:57,382 Epoch  58 Step:    24900 Batch Loss:     0.701790 Tokens per Sec:    15188, Lr: 0.000300\n",
            "2019-11-26 12:49:13,541 Epoch  58 Step:    25000 Batch Loss:     1.179395 Tokens per Sec:    15134, Lr: 0.000300\n",
            "2019-11-26 12:49:29,471 Epoch  58 Step:    25100 Batch Loss:     0.944794 Tokens per Sec:    14946, Lr: 0.000300\n",
            "2019-11-26 12:49:45,669 Epoch  58 Step:    25200 Batch Loss:     0.907928 Tokens per Sec:    15309, Lr: 0.000300\n",
            "2019-11-26 12:50:01,698 Epoch  58 Step:    25300 Batch Loss:     1.173214 Tokens per Sec:    15466, Lr: 0.000300\n",
            "2019-11-26 12:50:04,882 Epoch  58: total training loss 384.27\n",
            "2019-11-26 12:50:04,882 EPOCH 59\n",
            "2019-11-26 12:50:17,875 Epoch  59 Step:    25400 Batch Loss:     0.998782 Tokens per Sec:    15550, Lr: 0.000300\n",
            "2019-11-26 12:50:33,865 Epoch  59 Step:    25500 Batch Loss:     1.223902 Tokens per Sec:    15178, Lr: 0.000300\n",
            "2019-11-26 12:50:49,993 Epoch  59 Step:    25600 Batch Loss:     1.320425 Tokens per Sec:    15353, Lr: 0.000300\n",
            "2019-11-26 12:51:06,125 Epoch  59 Step:    25700 Batch Loss:     0.623917 Tokens per Sec:    15074, Lr: 0.000300\n",
            "2019-11-26 12:51:15,376 Epoch  59: total training loss 380.87\n",
            "2019-11-26 12:51:15,376 EPOCH 60\n",
            "2019-11-26 12:51:22,285 Epoch  60 Step:    25800 Batch Loss:     1.006226 Tokens per Sec:    15034, Lr: 0.000300\n",
            "2019-11-26 12:51:38,694 Epoch  60 Step:    25900 Batch Loss:     0.843723 Tokens per Sec:    14957, Lr: 0.000300\n",
            "2019-11-26 12:51:54,693 Epoch  60 Step:    26000 Batch Loss:     0.444974 Tokens per Sec:    15440, Lr: 0.000300\n",
            "2019-11-26 12:52:43,820 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:52:43,820 Saving new checkpoint.\n",
            "2019-11-26 12:52:44,111 Example #0\n",
            "2019-11-26 12:52:44,112 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:52:44,112 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:52:44,112 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:52:44,112 Example #1\n",
            "2019-11-26 12:52:44,112 \tSource:     The Drostdy\n",
            "2019-11-26 12:52:44,112 \tReference:  Die Drostdy\n",
            "2019-11-26 12:52:44,112 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:52:44,113 Example #2\n",
            "2019-11-26 12:52:44,113 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:52:44,113 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:52:44,113 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoering van persone buite die kommissie te verplig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:52:44,113 Example #3\n",
            "2019-11-26 12:52:44,113 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:52:44,113 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:52:44,114 \tHypothesis: * private aktiwiteite deur individue\n",
            "2019-11-26 12:52:44,114 Validation result (greedy) at epoch  60, step    26000: bleu:  47.45, loss: 26914.4805, ppl:   3.1692, duration: 49.4213s\n",
            "2019-11-26 12:53:00,273 Epoch  60 Step:    26100 Batch Loss:     0.989446 Tokens per Sec:    15227, Lr: 0.000300\n",
            "2019-11-26 12:53:15,584 Epoch  60: total training loss 376.27\n",
            "2019-11-26 12:53:15,585 EPOCH 61\n",
            "2019-11-26 12:53:16,397 Epoch  61 Step:    26200 Batch Loss:     0.575541 Tokens per Sec:    12867, Lr: 0.000300\n",
            "2019-11-26 12:53:32,510 Epoch  61 Step:    26300 Batch Loss:     0.673162 Tokens per Sec:    15553, Lr: 0.000300\n",
            "2019-11-26 12:53:48,457 Epoch  61 Step:    26400 Batch Loss:     0.687498 Tokens per Sec:    15051, Lr: 0.000300\n",
            "2019-11-26 12:54:04,736 Epoch  61 Step:    26500 Batch Loss:     1.163097 Tokens per Sec:    15211, Lr: 0.000300\n",
            "2019-11-26 12:54:20,588 Epoch  61 Step:    26600 Batch Loss:     0.736466 Tokens per Sec:    14963, Lr: 0.000300\n",
            "2019-11-26 12:54:26,167 Epoch  61: total training loss 376.97\n",
            "2019-11-26 12:54:26,167 EPOCH 62\n",
            "2019-11-26 12:54:36,973 Epoch  62 Step:    26700 Batch Loss:     0.550314 Tokens per Sec:    15221, Lr: 0.000300\n",
            "2019-11-26 12:54:52,867 Epoch  62 Step:    26800 Batch Loss:     1.064360 Tokens per Sec:    15105, Lr: 0.000300\n",
            "2019-11-26 12:55:08,788 Epoch  62 Step:    26900 Batch Loss:     1.218435 Tokens per Sec:    15168, Lr: 0.000300\n",
            "2019-11-26 12:55:24,851 Epoch  62 Step:    27000 Batch Loss:     1.178409 Tokens per Sec:    15256, Lr: 0.000300\n",
            "2019-11-26 12:55:36,683 Epoch  62: total training loss 373.81\n",
            "2019-11-26 12:55:36,683 EPOCH 63\n",
            "2019-11-26 12:55:41,018 Epoch  63 Step:    27100 Batch Loss:     0.433264 Tokens per Sec:    14612, Lr: 0.000300\n",
            "2019-11-26 12:55:57,274 Epoch  63 Step:    27200 Batch Loss:     0.690832 Tokens per Sec:    15475, Lr: 0.000300\n",
            "2019-11-26 12:56:13,298 Epoch  63 Step:    27300 Batch Loss:     1.054951 Tokens per Sec:    15144, Lr: 0.000300\n",
            "2019-11-26 12:56:29,503 Epoch  63 Step:    27400 Batch Loss:     0.663507 Tokens per Sec:    15662, Lr: 0.000300\n",
            "2019-11-26 12:56:45,325 Epoch  63 Step:    27500 Batch Loss:     0.971017 Tokens per Sec:    14988, Lr: 0.000300\n",
            "2019-11-26 12:56:46,909 Epoch  63: total training loss 367.83\n",
            "2019-11-26 12:56:46,909 EPOCH 64\n",
            "2019-11-26 12:57:01,375 Epoch  64 Step:    27600 Batch Loss:     1.309990 Tokens per Sec:    15379, Lr: 0.000300\n",
            "2019-11-26 12:57:17,300 Epoch  64 Step:    27700 Batch Loss:     0.708928 Tokens per Sec:    15200, Lr: 0.000300\n",
            "2019-11-26 12:57:33,597 Epoch  64 Step:    27800 Batch Loss:     0.759926 Tokens per Sec:    15742, Lr: 0.000300\n",
            "2019-11-26 12:57:49,383 Epoch  64 Step:    27900 Batch Loss:     0.767020 Tokens per Sec:    15023, Lr: 0.000300\n",
            "2019-11-26 12:57:56,928 Epoch  64: total training loss 365.45\n",
            "2019-11-26 12:57:56,928 EPOCH 65\n",
            "2019-11-26 12:58:05,349 Epoch  65 Step:    28000 Batch Loss:     1.017252 Tokens per Sec:    15326, Lr: 0.000300\n",
            "2019-11-26 12:58:54,663 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 12:58:54,663 Saving new checkpoint.\n",
            "2019-11-26 12:58:54,962 Example #0\n",
            "2019-11-26 12:58:54,962 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 12:58:54,962 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:58:54,962 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 12:58:54,962 Example #1\n",
            "2019-11-26 12:58:54,962 \tSource:     The Drostdy\n",
            "2019-11-26 12:58:54,962 \tReference:  Die Drostdy\n",
            "2019-11-26 12:58:54,962 \tHypothesis: Die Drostdy\n",
            "2019-11-26 12:58:54,962 Example #2\n",
            "2019-11-26 12:58:54,963 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 12:58:54,963 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 12:58:54,963 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoering van persone buite die kommissie te verplig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 12:58:54,963 Example #3\n",
            "2019-11-26 12:58:54,963 \tSource:     * private acts by individuals\n",
            "2019-11-26 12:58:54,963 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 12:58:54,963 \tHypothesis: * private aktiwiteite deur individue\n",
            "2019-11-26 12:58:54,963 Validation result (greedy) at epoch  65, step    28000: bleu:  47.00, loss: 26760.4766, ppl:   3.1484, duration: 49.6141s\n",
            "2019-11-26 12:59:11,290 Epoch  65 Step:    28100 Batch Loss:     0.542845 Tokens per Sec:    14901, Lr: 0.000300\n",
            "2019-11-26 12:59:27,794 Epoch  65 Step:    28200 Batch Loss:     0.637733 Tokens per Sec:    14801, Lr: 0.000300\n",
            "2019-11-26 12:59:44,132 Epoch  65 Step:    28300 Batch Loss:     0.789681 Tokens per Sec:    15229, Lr: 0.000300\n",
            "2019-11-26 12:59:57,706 Epoch  65: total training loss 361.99\n",
            "2019-11-26 12:59:57,706 EPOCH 66\n",
            "2019-11-26 13:00:00,397 Epoch  66 Step:    28400 Batch Loss:     0.498018 Tokens per Sec:    15255, Lr: 0.000300\n",
            "2019-11-26 13:00:16,660 Epoch  66 Step:    28500 Batch Loss:     0.784142 Tokens per Sec:    15059, Lr: 0.000300\n",
            "2019-11-26 13:00:32,896 Epoch  66 Step:    28600 Batch Loss:     1.109083 Tokens per Sec:    15053, Lr: 0.000300\n",
            "2019-11-26 13:00:49,245 Epoch  66 Step:    28700 Batch Loss:     0.537676 Tokens per Sec:    15134, Lr: 0.000300\n",
            "2019-11-26 13:01:05,438 Epoch  66 Step:    28800 Batch Loss:     0.826802 Tokens per Sec:    15258, Lr: 0.000300\n",
            "2019-11-26 13:01:08,645 Epoch  66: total training loss 357.63\n",
            "2019-11-26 13:01:08,645 EPOCH 67\n",
            "2019-11-26 13:01:21,714 Epoch  67 Step:    28900 Batch Loss:     0.987926 Tokens per Sec:    15226, Lr: 0.000300\n",
            "2019-11-26 13:01:38,055 Epoch  67 Step:    29000 Batch Loss:     0.745742 Tokens per Sec:    15080, Lr: 0.000300\n",
            "2019-11-26 13:01:54,296 Epoch  67 Step:    29100 Batch Loss:     0.377114 Tokens per Sec:    15214, Lr: 0.000300\n",
            "2019-11-26 13:02:10,495 Epoch  67 Step:    29200 Batch Loss:     0.678766 Tokens per Sec:    15277, Lr: 0.000300\n",
            "2019-11-26 13:02:19,452 Epoch  67: total training loss 355.98\n",
            "2019-11-26 13:02:19,452 EPOCH 68\n",
            "2019-11-26 13:02:26,590 Epoch  68 Step:    29300 Batch Loss:     0.721982 Tokens per Sec:    15453, Lr: 0.000300\n",
            "2019-11-26 13:02:42,437 Epoch  68 Step:    29400 Batch Loss:     0.919434 Tokens per Sec:    14818, Lr: 0.000300\n",
            "2019-11-26 13:02:58,542 Epoch  68 Step:    29500 Batch Loss:     0.370095 Tokens per Sec:    15163, Lr: 0.000300\n",
            "2019-11-26 13:03:15,036 Epoch  68 Step:    29600 Batch Loss:     0.557850 Tokens per Sec:    15233, Lr: 0.000300\n",
            "2019-11-26 13:03:30,402 Epoch  68: total training loss 354.75\n",
            "2019-11-26 13:03:30,403 EPOCH 69\n",
            "2019-11-26 13:03:31,349 Epoch  69 Step:    29700 Batch Loss:     0.982612 Tokens per Sec:    15084, Lr: 0.000300\n",
            "2019-11-26 13:03:47,578 Epoch  69 Step:    29800 Batch Loss:     0.917254 Tokens per Sec:    15089, Lr: 0.000300\n",
            "2019-11-26 13:04:03,805 Epoch  69 Step:    29900 Batch Loss:     0.886195 Tokens per Sec:    15005, Lr: 0.000300\n",
            "2019-11-26 13:04:19,939 Epoch  69 Step:    30000 Batch Loss:     0.775156 Tokens per Sec:    15081, Lr: 0.000300\n",
            "2019-11-26 13:05:09,194 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:05:09,194 Saving new checkpoint.\n",
            "2019-11-26 13:05:09,494 Example #0\n",
            "2019-11-26 13:05:09,494 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:05:09,494 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:05:09,494 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:05:09,494 Example #1\n",
            "2019-11-26 13:05:09,495 \tSource:     The Drostdy\n",
            "2019-11-26 13:05:09,495 \tReference:  Die Drostdy\n",
            "2019-11-26 13:05:09,495 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:05:09,495 Example #2\n",
            "2019-11-26 13:05:09,495 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:05:09,495 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:05:09,495 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoer van persone buite die kommissie te verrig en te verseker dat gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie betrokke is .\n",
            "2019-11-26 13:05:09,495 Example #3\n",
            "2019-11-26 13:05:09,495 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:05:09,495 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:05:09,495 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:05:09,495 Validation result (greedy) at epoch  69, step    30000: bleu:  47.77, loss: 26577.5254, ppl:   3.1238, duration: 49.5561s\n",
            "2019-11-26 13:05:25,880 Epoch  69 Step:    30100 Batch Loss:     0.887959 Tokens per Sec:    15364, Lr: 0.000300\n",
            "2019-11-26 13:05:30,743 Epoch  69: total training loss 350.43\n",
            "2019-11-26 13:05:30,743 EPOCH 70\n",
            "2019-11-26 13:05:41,846 Epoch  70 Step:    30200 Batch Loss:     0.631063 Tokens per Sec:    14963, Lr: 0.000300\n",
            "2019-11-26 13:05:58,055 Epoch  70 Step:    30300 Batch Loss:     0.611507 Tokens per Sec:    15184, Lr: 0.000300\n",
            "2019-11-26 13:06:14,201 Epoch  70 Step:    30400 Batch Loss:     0.539814 Tokens per Sec:    15244, Lr: 0.000300\n",
            "2019-11-26 13:06:30,591 Epoch  70 Step:    30500 Batch Loss:     1.071048 Tokens per Sec:    15352, Lr: 0.000300\n",
            "2019-11-26 13:06:41,451 Epoch  70: total training loss 351.13\n",
            "2019-11-26 13:06:41,451 EPOCH 71\n",
            "2019-11-26 13:06:46,726 Epoch  71 Step:    30600 Batch Loss:     0.711578 Tokens per Sec:    15386, Lr: 0.000300\n",
            "2019-11-26 13:07:02,862 Epoch  71 Step:    30700 Batch Loss:     0.814951 Tokens per Sec:    14923, Lr: 0.000300\n",
            "2019-11-26 13:07:19,091 Epoch  71 Step:    30800 Batch Loss:     1.108014 Tokens per Sec:    15314, Lr: 0.000300\n",
            "2019-11-26 13:07:35,279 Epoch  71 Step:    30900 Batch Loss:     0.923361 Tokens per Sec:    15218, Lr: 0.000300\n",
            "2019-11-26 13:07:51,504 Epoch  71 Step:    31000 Batch Loss:     0.925191 Tokens per Sec:    14991, Lr: 0.000300\n",
            "2019-11-26 13:07:52,322 Epoch  71: total training loss 348.45\n",
            "2019-11-26 13:07:52,322 EPOCH 72\n",
            "2019-11-26 13:08:07,849 Epoch  72 Step:    31100 Batch Loss:     1.060984 Tokens per Sec:    15239, Lr: 0.000300\n",
            "2019-11-26 13:08:23,939 Epoch  72 Step:    31200 Batch Loss:     0.665788 Tokens per Sec:    15055, Lr: 0.000300\n",
            "2019-11-26 13:08:40,298 Epoch  72 Step:    31300 Batch Loss:     1.167653 Tokens per Sec:    15369, Lr: 0.000300\n",
            "2019-11-26 13:08:56,488 Epoch  72 Step:    31400 Batch Loss:     0.965262 Tokens per Sec:    14999, Lr: 0.000300\n",
            "2019-11-26 13:09:03,006 Epoch  72: total training loss 342.94\n",
            "2019-11-26 13:09:03,006 EPOCH 73\n",
            "2019-11-26 13:09:12,876 Epoch  73 Step:    31500 Batch Loss:     1.073630 Tokens per Sec:    15205, Lr: 0.000300\n",
            "2019-11-26 13:09:29,245 Epoch  73 Step:    31600 Batch Loss:     0.324575 Tokens per Sec:    15429, Lr: 0.000300\n",
            "2019-11-26 13:09:45,104 Epoch  73 Step:    31700 Batch Loss:     1.072775 Tokens per Sec:    14945, Lr: 0.000300\n",
            "2019-11-26 13:10:01,039 Epoch  73 Step:    31800 Batch Loss:     0.528711 Tokens per Sec:    14972, Lr: 0.000300\n",
            "2019-11-26 13:10:13,539 Epoch  73: total training loss 342.31\n",
            "2019-11-26 13:10:13,540 EPOCH 74\n",
            "2019-11-26 13:10:17,500 Epoch  74 Step:    31900 Batch Loss:     0.875707 Tokens per Sec:    14605, Lr: 0.000300\n",
            "2019-11-26 13:10:33,860 Epoch  74 Step:    32000 Batch Loss:     0.853001 Tokens per Sec:    15452, Lr: 0.000300\n",
            "2019-11-26 13:11:23,031 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:11:23,031 Saving new checkpoint.\n",
            "2019-11-26 13:11:23,334 Example #0\n",
            "2019-11-26 13:11:23,335 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:11:23,335 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:11:23,335 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:11:23,335 Example #1\n",
            "2019-11-26 13:11:23,335 \tSource:     The Drostdy\n",
            "2019-11-26 13:11:23,335 \tReference:  Die Drostdy\n",
            "2019-11-26 13:11:23,335 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:11:23,336 Example #2\n",
            "2019-11-26 13:11:23,336 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:11:23,336 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:11:23,336 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoer van persone buite die kommissie te verplig en om gemeenskapsbetrokkenheid in die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 13:11:23,336 Example #3\n",
            "2019-11-26 13:11:23,336 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:11:23,336 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:11:23,336 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:11:23,336 Validation result (greedy) at epoch  74, step    32000: bleu:  49.18, loss: 26218.0801, ppl:   3.0761, duration: 49.4765s\n",
            "2019-11-26 13:11:39,405 Epoch  74 Step:    32100 Batch Loss:     0.682031 Tokens per Sec:    15277, Lr: 0.000300\n",
            "2019-11-26 13:11:55,492 Epoch  74 Step:    32200 Batch Loss:     0.591827 Tokens per Sec:    15119, Lr: 0.000300\n",
            "2019-11-26 13:12:11,506 Epoch  74 Step:    32300 Batch Loss:     1.013833 Tokens per Sec:    15392, Lr: 0.000300\n",
            "2019-11-26 13:12:13,305 Epoch  74: total training loss 337.46\n",
            "2019-11-26 13:12:13,306 EPOCH 75\n",
            "2019-11-26 13:12:27,542 Epoch  75 Step:    32400 Batch Loss:     1.147855 Tokens per Sec:    15091, Lr: 0.000300\n",
            "2019-11-26 13:12:43,678 Epoch  75 Step:    32500 Batch Loss:     0.578004 Tokens per Sec:    15168, Lr: 0.000300\n",
            "2019-11-26 13:12:59,743 Epoch  75 Step:    32600 Batch Loss:     0.668102 Tokens per Sec:    15015, Lr: 0.000300\n",
            "2019-11-26 13:13:15,971 Epoch  75 Step:    32700 Batch Loss:     0.725984 Tokens per Sec:    15305, Lr: 0.000300\n",
            "2019-11-26 13:13:23,987 Epoch  75: total training loss 338.22\n",
            "2019-11-26 13:13:23,988 EPOCH 76\n",
            "2019-11-26 13:13:32,069 Epoch  76 Step:    32800 Batch Loss:     0.544484 Tokens per Sec:    15270, Lr: 0.000300\n",
            "2019-11-26 13:13:48,121 Epoch  76 Step:    32900 Batch Loss:     0.547336 Tokens per Sec:    15005, Lr: 0.000300\n",
            "2019-11-26 13:14:04,291 Epoch  76 Step:    33000 Batch Loss:     1.006842 Tokens per Sec:    15109, Lr: 0.000300\n",
            "2019-11-26 13:14:20,384 Epoch  76 Step:    33100 Batch Loss:     0.723646 Tokens per Sec:    15522, Lr: 0.000300\n",
            "2019-11-26 13:14:34,480 Epoch  76: total training loss 336.95\n",
            "2019-11-26 13:14:34,480 EPOCH 77\n",
            "2019-11-26 13:14:36,548 Epoch  77 Step:    33200 Batch Loss:     1.025953 Tokens per Sec:    15535, Lr: 0.000300\n",
            "2019-11-26 13:14:52,589 Epoch  77 Step:    33300 Batch Loss:     1.019696 Tokens per Sec:    15200, Lr: 0.000300\n",
            "2019-11-26 13:15:08,822 Epoch  77 Step:    33400 Batch Loss:     0.669594 Tokens per Sec:    15209, Lr: 0.000300\n",
            "2019-11-26 13:15:24,993 Epoch  77 Step:    33500 Batch Loss:     0.671676 Tokens per Sec:    15374, Lr: 0.000300\n",
            "2019-11-26 13:15:40,984 Epoch  77 Step:    33600 Batch Loss:     0.650119 Tokens per Sec:    15199, Lr: 0.000300\n",
            "2019-11-26 13:15:44,736 Epoch  77: total training loss 332.27\n",
            "2019-11-26 13:15:44,736 EPOCH 78\n",
            "2019-11-26 13:15:57,337 Epoch  78 Step:    33700 Batch Loss:     0.544939 Tokens per Sec:    15133, Lr: 0.000300\n",
            "2019-11-26 13:16:13,560 Epoch  78 Step:    33800 Batch Loss:     0.922034 Tokens per Sec:    15277, Lr: 0.000300\n",
            "2019-11-26 13:16:29,537 Epoch  78 Step:    33900 Batch Loss:     1.048963 Tokens per Sec:    15282, Lr: 0.000300\n",
            "2019-11-26 13:16:45,623 Epoch  78 Step:    34000 Batch Loss:     0.721920 Tokens per Sec:    15424, Lr: 0.000300\n",
            "2019-11-26 13:17:34,750 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:17:34,750 Saving new checkpoint.\n",
            "2019-11-26 13:17:35,073 Example #0\n",
            "2019-11-26 13:17:35,073 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:17:35,073 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:17:35,073 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:17:35,073 Example #1\n",
            "2019-11-26 13:17:35,073 \tSource:     The Drostdy\n",
            "2019-11-26 13:17:35,074 \tReference:  Die Drostdy\n",
            "2019-11-26 13:17:35,074 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:17:35,074 Example #2\n",
            "2019-11-26 13:17:35,074 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:17:35,074 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:17:35,074 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die presiese kennis van persone buite die kommissie te verrig en om regstreekse betrokkenheid te verseker in die aktiwiteite van die kommissie .\n",
            "2019-11-26 13:17:35,074 Example #3\n",
            "2019-11-26 13:17:35,074 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:17:35,074 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:17:35,074 \tHypothesis: * private aktiwiteite deur individue\n",
            "2019-11-26 13:17:35,074 Validation result (greedy) at epoch  78, step    34000: bleu:  49.27, loss: 26035.1660, ppl:   3.0520, duration: 49.4508s\n",
            "2019-11-26 13:17:44,581 Epoch  78: total training loss 331.04\n",
            "2019-11-26 13:17:44,581 EPOCH 79\n",
            "2019-11-26 13:17:51,227 Epoch  79 Step:    34100 Batch Loss:     0.854058 Tokens per Sec:    14864, Lr: 0.000300\n",
            "2019-11-26 13:18:07,396 Epoch  79 Step:    34200 Batch Loss:     0.748328 Tokens per Sec:    15361, Lr: 0.000300\n",
            "2019-11-26 13:18:23,437 Epoch  79 Step:    34300 Batch Loss:     0.907164 Tokens per Sec:    14928, Lr: 0.000300\n",
            "2019-11-26 13:18:39,561 Epoch  79 Step:    34400 Batch Loss:     0.965993 Tokens per Sec:    15413, Lr: 0.000300\n",
            "2019-11-26 13:18:54,879 Epoch  79: total training loss 328.70\n",
            "2019-11-26 13:18:54,879 EPOCH 80\n",
            "2019-11-26 13:18:55,753 Epoch  80 Step:    34500 Batch Loss:     1.017356 Tokens per Sec:    12591, Lr: 0.000300\n",
            "2019-11-26 13:19:11,952 Epoch  80 Step:    34600 Batch Loss:     0.848761 Tokens per Sec:    15422, Lr: 0.000300\n",
            "2019-11-26 13:19:28,000 Epoch  80 Step:    34700 Batch Loss:     0.715679 Tokens per Sec:    15278, Lr: 0.000300\n",
            "2019-11-26 13:19:43,964 Epoch  80 Step:    34800 Batch Loss:     0.535557 Tokens per Sec:    14932, Lr: 0.000300\n",
            "2019-11-26 13:19:59,921 Epoch  80 Step:    34900 Batch Loss:     0.861750 Tokens per Sec:    15064, Lr: 0.000300\n",
            "2019-11-26 13:20:05,480 Epoch  80: total training loss 329.55\n",
            "2019-11-26 13:20:05,480 EPOCH 81\n",
            "2019-11-26 13:20:16,149 Epoch  81 Step:    35000 Batch Loss:     1.011204 Tokens per Sec:    15059, Lr: 0.000300\n",
            "2019-11-26 13:20:32,245 Epoch  81 Step:    35100 Batch Loss:     0.744987 Tokens per Sec:    14999, Lr: 0.000300\n",
            "2019-11-26 13:20:48,366 Epoch  81 Step:    35200 Batch Loss:     0.741128 Tokens per Sec:    15498, Lr: 0.000300\n",
            "2019-11-26 13:21:04,459 Epoch  81 Step:    35300 Batch Loss:     0.439612 Tokens per Sec:    15167, Lr: 0.000300\n",
            "2019-11-26 13:21:15,929 Epoch  81: total training loss 324.50\n",
            "2019-11-26 13:21:15,929 EPOCH 82\n",
            "2019-11-26 13:21:20,728 Epoch  82 Step:    35400 Batch Loss:     0.640325 Tokens per Sec:    14950, Lr: 0.000300\n",
            "2019-11-26 13:21:36,610 Epoch  82 Step:    35500 Batch Loss:     0.623967 Tokens per Sec:    14980, Lr: 0.000300\n",
            "2019-11-26 13:21:52,849 Epoch  82 Step:    35600 Batch Loss:     0.806691 Tokens per Sec:    15303, Lr: 0.000300\n",
            "2019-11-26 13:22:09,099 Epoch  82 Step:    35700 Batch Loss:     0.562056 Tokens per Sec:    15388, Lr: 0.000300\n",
            "2019-11-26 13:22:25,086 Epoch  82 Step:    35800 Batch Loss:     0.864480 Tokens per Sec:    15084, Lr: 0.000300\n",
            "2019-11-26 13:22:26,565 Epoch  82: total training loss 323.27\n",
            "2019-11-26 13:22:26,566 EPOCH 83\n",
            "2019-11-26 13:22:41,166 Epoch  83 Step:    35900 Batch Loss:     0.389210 Tokens per Sec:    15278, Lr: 0.000300\n",
            "2019-11-26 13:22:57,289 Epoch  83 Step:    36000 Batch Loss:     0.587104 Tokens per Sec:    15225, Lr: 0.000300\n",
            "2019-11-26 13:23:46,417 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:23:46,418 Saving new checkpoint.\n",
            "2019-11-26 13:23:46,719 Example #0\n",
            "2019-11-26 13:23:46,720 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:23:46,720 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:23:46,720 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:23:46,720 Example #1\n",
            "2019-11-26 13:23:46,720 \tSource:     The Drostdy\n",
            "2019-11-26 13:23:46,720 \tReference:  Die Drostdy\n",
            "2019-11-26 13:23:46,720 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:23:46,720 Example #2\n",
            "2019-11-26 13:23:46,720 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:23:46,720 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:23:46,720 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die kundigheid van persone buite die kommissie te verrig en om gemeenskapsbetrokkenheid te verseker in die aktiwiteite van die kommissie .\n",
            "2019-11-26 13:23:46,720 Example #3\n",
            "2019-11-26 13:23:46,721 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:23:46,721 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:23:46,721 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:23:46,721 Validation result (greedy) at epoch  83, step    36000: bleu:  49.67, loss: 25899.0938, ppl:   3.0343, duration: 49.4310s\n",
            "2019-11-26 13:24:03,098 Epoch  83 Step:    36100 Batch Loss:     0.924729 Tokens per Sec:    15170, Lr: 0.000300\n",
            "2019-11-26 13:24:19,307 Epoch  83 Step:    36200 Batch Loss:     0.973561 Tokens per Sec:    14914, Lr: 0.000300\n",
            "2019-11-26 13:24:26,586 Epoch  83: total training loss 319.50\n",
            "2019-11-26 13:24:26,586 EPOCH 84\n",
            "2019-11-26 13:24:35,572 Epoch  84 Step:    36300 Batch Loss:     0.934725 Tokens per Sec:    15152, Lr: 0.000300\n",
            "2019-11-26 13:24:51,748 Epoch  84 Step:    36400 Batch Loss:     0.762359 Tokens per Sec:    15118, Lr: 0.000300\n",
            "2019-11-26 13:25:07,966 Epoch  84 Step:    36500 Batch Loss:     0.696560 Tokens per Sec:    15168, Lr: 0.000300\n",
            "2019-11-26 13:25:24,099 Epoch  84 Step:    36600 Batch Loss:     0.438995 Tokens per Sec:    15300, Lr: 0.000300\n",
            "2019-11-26 13:25:37,164 Epoch  84: total training loss 317.51\n",
            "2019-11-26 13:25:37,164 EPOCH 85\n",
            "2019-11-26 13:25:40,351 Epoch  85 Step:    36700 Batch Loss:     0.583036 Tokens per Sec:    14714, Lr: 0.000300\n",
            "2019-11-26 13:25:56,375 Epoch  85 Step:    36800 Batch Loss:     0.678274 Tokens per Sec:    15186, Lr: 0.000300\n",
            "2019-11-26 13:26:12,684 Epoch  85 Step:    36900 Batch Loss:     0.599494 Tokens per Sec:    15311, Lr: 0.000300\n",
            "2019-11-26 13:26:28,685 Epoch  85 Step:    37000 Batch Loss:     1.023514 Tokens per Sec:    15295, Lr: 0.000300\n",
            "2019-11-26 13:26:45,006 Epoch  85 Step:    37100 Batch Loss:     0.721533 Tokens per Sec:    15170, Lr: 0.000300\n",
            "2019-11-26 13:26:47,724 Epoch  85: total training loss 316.44\n",
            "2019-11-26 13:26:47,724 EPOCH 86\n",
            "2019-11-26 13:27:01,176 Epoch  86 Step:    37200 Batch Loss:     0.839128 Tokens per Sec:    15268, Lr: 0.000300\n",
            "2019-11-26 13:27:17,346 Epoch  86 Step:    37300 Batch Loss:     0.734333 Tokens per Sec:    15305, Lr: 0.000300\n",
            "2019-11-26 13:27:33,374 Epoch  86 Step:    37400 Batch Loss:     0.508487 Tokens per Sec:    15344, Lr: 0.000300\n",
            "2019-11-26 13:27:49,688 Epoch  86 Step:    37500 Batch Loss:     1.090178 Tokens per Sec:    15320, Lr: 0.000300\n",
            "2019-11-26 13:27:57,890 Epoch  86: total training loss 313.48\n",
            "2019-11-26 13:27:57,890 EPOCH 87\n",
            "2019-11-26 13:28:05,851 Epoch  87 Step:    37600 Batch Loss:     0.479971 Tokens per Sec:    15041, Lr: 0.000300\n",
            "2019-11-26 13:28:22,097 Epoch  87 Step:    37700 Batch Loss:     0.454109 Tokens per Sec:    15331, Lr: 0.000300\n",
            "2019-11-26 13:28:38,159 Epoch  87 Step:    37800 Batch Loss:     0.441764 Tokens per Sec:    15315, Lr: 0.000300\n",
            "2019-11-26 13:28:54,355 Epoch  87 Step:    37900 Batch Loss:     0.903597 Tokens per Sec:    15444, Lr: 0.000300\n",
            "2019-11-26 13:29:08,252 Epoch  87: total training loss 311.77\n",
            "2019-11-26 13:29:08,253 EPOCH 88\n",
            "2019-11-26 13:29:10,382 Epoch  88 Step:    38000 Batch Loss:     0.977019 Tokens per Sec:    14602, Lr: 0.000300\n",
            "2019-11-26 13:29:59,453 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:29:59,453 Saving new checkpoint.\n",
            "2019-11-26 13:29:59,765 Example #0\n",
            "2019-11-26 13:29:59,765 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:29:59,765 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:29:59,765 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:29:59,765 Example #1\n",
            "2019-11-26 13:29:59,766 \tSource:     The Drostdy\n",
            "2019-11-26 13:29:59,766 \tReference:  Die Drostdy\n",
            "2019-11-26 13:29:59,766 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:29:59,766 Example #2\n",
            "2019-11-26 13:29:59,766 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:29:59,766 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:29:59,766 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die kundigheid van persone buite die kommissie te verplig en om gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie te verseker .\n",
            "2019-11-26 13:29:59,766 Example #3\n",
            "2019-11-26 13:29:59,766 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:29:59,766 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:29:59,766 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:29:59,766 Validation result (greedy) at epoch  88, step    38000: bleu:  50.04, loss: 25706.0918, ppl:   3.0093, duration: 49.3840s\n",
            "2019-11-26 13:30:16,101 Epoch  88 Step:    38100 Batch Loss:     0.665654 Tokens per Sec:    15339, Lr: 0.000300\n",
            "2019-11-26 13:30:32,225 Epoch  88 Step:    38200 Batch Loss:     0.896674 Tokens per Sec:    15138, Lr: 0.000300\n",
            "2019-11-26 13:30:48,275 Epoch  88 Step:    38300 Batch Loss:     0.852050 Tokens per Sec:    15208, Lr: 0.000300\n",
            "2019-11-26 13:31:04,404 Epoch  88 Step:    38400 Batch Loss:     0.578186 Tokens per Sec:    15370, Lr: 0.000300\n",
            "2019-11-26 13:31:07,957 Epoch  88: total training loss 309.43\n",
            "2019-11-26 13:31:07,957 EPOCH 89\n",
            "2019-11-26 13:31:20,442 Epoch  89 Step:    38500 Batch Loss:     0.734052 Tokens per Sec:    14887, Lr: 0.000300\n",
            "2019-11-26 13:31:36,694 Epoch  89 Step:    38600 Batch Loss:     0.467864 Tokens per Sec:    15335, Lr: 0.000300\n",
            "2019-11-26 13:31:52,862 Epoch  89 Step:    38700 Batch Loss:     0.987611 Tokens per Sec:    15180, Lr: 0.000300\n",
            "2019-11-26 13:32:08,826 Epoch  89 Step:    38800 Batch Loss:     0.947608 Tokens per Sec:    15310, Lr: 0.000300\n",
            "2019-11-26 13:32:18,337 Epoch  89: total training loss 309.07\n",
            "2019-11-26 13:32:18,337 EPOCH 90\n",
            "2019-11-26 13:32:24,927 Epoch  90 Step:    38900 Batch Loss:     0.760241 Tokens per Sec:    14917, Lr: 0.000300\n",
            "2019-11-26 13:32:41,115 Epoch  90 Step:    39000 Batch Loss:     0.988469 Tokens per Sec:    15270, Lr: 0.000300\n",
            "2019-11-26 13:32:57,083 Epoch  90 Step:    39100 Batch Loss:     0.380122 Tokens per Sec:    15434, Lr: 0.000300\n",
            "2019-11-26 13:33:13,302 Epoch  90 Step:    39200 Batch Loss:     0.398116 Tokens per Sec:    15147, Lr: 0.000300\n",
            "2019-11-26 13:33:28,802 Epoch  90: total training loss 309.41\n",
            "2019-11-26 13:33:28,802 EPOCH 91\n",
            "2019-11-26 13:33:29,346 Epoch  91 Step:    39300 Batch Loss:     0.462888 Tokens per Sec:    12529, Lr: 0.000300\n",
            "2019-11-26 13:33:45,464 Epoch  91 Step:    39400 Batch Loss:     0.643982 Tokens per Sec:    15392, Lr: 0.000300\n",
            "2019-11-26 13:34:01,637 Epoch  91 Step:    39500 Batch Loss:     1.007519 Tokens per Sec:    15314, Lr: 0.000300\n",
            "2019-11-26 13:34:17,730 Epoch  91 Step:    39600 Batch Loss:     0.753260 Tokens per Sec:    15602, Lr: 0.000300\n",
            "2019-11-26 13:34:33,674 Epoch  91 Step:    39700 Batch Loss:     0.654618 Tokens per Sec:    15012, Lr: 0.000300\n",
            "2019-11-26 13:34:38,890 Epoch  91: total training loss 303.97\n",
            "2019-11-26 13:34:38,890 EPOCH 92\n",
            "2019-11-26 13:34:49,837 Epoch  92 Step:    39800 Batch Loss:     0.793661 Tokens per Sec:    15403, Lr: 0.000300\n",
            "2019-11-26 13:35:05,771 Epoch  92 Step:    39900 Batch Loss:     0.650563 Tokens per Sec:    15366, Lr: 0.000300\n",
            "2019-11-26 13:35:21,729 Epoch  92 Step:    40000 Batch Loss:     0.688826 Tokens per Sec:    15343, Lr: 0.000300\n",
            "2019-11-26 13:36:10,737 Hooray! New best validation result [ppl]!\n",
            "2019-11-26 13:36:10,737 Saving new checkpoint.\n",
            "2019-11-26 13:36:11,026 Example #0\n",
            "2019-11-26 13:36:11,027 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:36:11,027 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:36:11,027 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:36:11,027 Example #1\n",
            "2019-11-26 13:36:11,027 \tSource:     The Drostdy\n",
            "2019-11-26 13:36:11,027 \tReference:  Die Drostdy\n",
            "2019-11-26 13:36:11,027 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:36:11,027 Example #2\n",
            "2019-11-26 13:36:11,027 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:36:11,027 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:36:11,027 \tHypothesis: die oogmerk van die tweede kategorie van komitees is om die uitvoering van persone buite die kommissie te verrig en te verseker dat direkte gemeenskap betrokkenheid by die aktiwiteite van die kommissie .\n",
            "2019-11-26 13:36:11,027 Example #3\n",
            "2019-11-26 13:36:11,028 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:36:11,028 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:36:11,028 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:36:11,028 Validation result (greedy) at epoch  92, step    40000: bleu:  49.61, loss: 25581.3008, ppl:   2.9932, duration: 49.2989s\n",
            "2019-11-26 13:36:26,921 Epoch  92 Step:    40100 Batch Loss:     0.821566 Tokens per Sec:    15279, Lr: 0.000300\n",
            "2019-11-26 13:36:38,111 Epoch  92: total training loss 304.97\n",
            "2019-11-26 13:36:38,111 EPOCH 93\n",
            "2019-11-26 13:36:42,896 Epoch  93 Step:    40200 Batch Loss:     0.737165 Tokens per Sec:    15286, Lr: 0.000300\n",
            "2019-11-26 13:36:58,975 Epoch  93 Step:    40300 Batch Loss:     0.605476 Tokens per Sec:    15240, Lr: 0.000300\n",
            "2019-11-26 13:37:15,308 Epoch  93 Step:    40400 Batch Loss:     0.927479 Tokens per Sec:    15439, Lr: 0.000300\n",
            "2019-11-26 13:37:31,097 Epoch  93 Step:    40500 Batch Loss:     0.983400 Tokens per Sec:    15036, Lr: 0.000300\n",
            "2019-11-26 13:37:47,240 Epoch  93 Step:    40600 Batch Loss:     0.927685 Tokens per Sec:    15485, Lr: 0.000300\n",
            "2019-11-26 13:37:48,166 Epoch  93: total training loss 302.69\n",
            "2019-11-26 13:37:48,166 EPOCH 94\n",
            "2019-11-26 13:38:03,364 Epoch  94 Step:    40700 Batch Loss:     0.695183 Tokens per Sec:    15284, Lr: 0.000300\n",
            "2019-11-26 13:38:19,282 Epoch  94 Step:    40800 Batch Loss:     0.730492 Tokens per Sec:    15100, Lr: 0.000300\n",
            "2019-11-26 13:38:35,538 Epoch  94 Step:    40900 Batch Loss:     0.961166 Tokens per Sec:    15156, Lr: 0.000300\n",
            "2019-11-26 13:38:51,562 Epoch  94 Step:    41000 Batch Loss:     0.710943 Tokens per Sec:    15475, Lr: 0.000300\n",
            "2019-11-26 13:38:58,318 Epoch  94: total training loss 301.28\n",
            "2019-11-26 13:38:58,318 EPOCH 95\n",
            "2019-11-26 13:39:07,656 Epoch  95 Step:    41100 Batch Loss:     0.635635 Tokens per Sec:    15097, Lr: 0.000300\n",
            "2019-11-26 13:39:23,746 Epoch  95 Step:    41200 Batch Loss:     0.764734 Tokens per Sec:    15378, Lr: 0.000300\n",
            "2019-11-26 13:39:39,585 Epoch  95 Step:    41300 Batch Loss:     0.726223 Tokens per Sec:    14994, Lr: 0.000300\n",
            "2019-11-26 13:39:55,803 Epoch  95 Step:    41400 Batch Loss:     0.973904 Tokens per Sec:    15543, Lr: 0.000300\n",
            "2019-11-26 13:40:08,295 Epoch  95: total training loss 301.23\n",
            "2019-11-26 13:40:08,295 EPOCH 96\n",
            "2019-11-26 13:40:11,983 Epoch  96 Step:    41500 Batch Loss:     0.952833 Tokens per Sec:    15587, Lr: 0.000300\n",
            "2019-11-26 13:40:28,039 Epoch  96 Step:    41600 Batch Loss:     0.685765 Tokens per Sec:    15345, Lr: 0.000300\n",
            "2019-11-26 13:40:44,453 Epoch  96 Step:    41700 Batch Loss:     1.019803 Tokens per Sec:    15659, Lr: 0.000300\n",
            "2019-11-26 13:41:00,560 Epoch  96 Step:    41800 Batch Loss:     0.601478 Tokens per Sec:    15283, Lr: 0.000300\n",
            "2019-11-26 13:41:16,371 Epoch  96 Step:    41900 Batch Loss:     0.636775 Tokens per Sec:    14926, Lr: 0.000300\n",
            "2019-11-26 13:41:18,267 Epoch  96: total training loss 295.91\n",
            "2019-11-26 13:41:18,267 EPOCH 97\n",
            "2019-11-26 13:41:32,278 Epoch  97 Step:    42000 Batch Loss:     0.967375 Tokens per Sec:    15130, Lr: 0.000300\n",
            "2019-11-26 13:42:21,208 Example #0\n",
            "2019-11-26 13:42:21,208 \tSource:     GEORGE LOCAL MUNICIPALITY: CONTACT\n",
            "2019-11-26 13:42:21,208 \tReference:  GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:42:21,208 \tHypothesis: GEORGE PLAASLIKE MUNISIPALITEIT: KONTAK\n",
            "2019-11-26 13:42:21,208 Example #1\n",
            "2019-11-26 13:42:21,209 \tSource:     The Drostdy\n",
            "2019-11-26 13:42:21,209 \tReference:  Die Drostdy\n",
            "2019-11-26 13:42:21,209 \tHypothesis: Die Drostdy\n",
            "2019-11-26 13:42:21,209 Example #2\n",
            "2019-11-26 13:42:21,209 \tSource:     the object of the second category of committees is to utilise the expertise of persons outside the commission and to ensure direct community involvement in the activities of the commission .\n",
            "2019-11-26 13:42:21,209 \tReference:  die doel van die tweede kategorie van komitees is om ook die kundigheid van persone buite die kommissie vir die kommissie se werksaamhede te benut en om direkte gemeenskapsbetrokkenheid te verseker .\n",
            "2019-11-26 13:42:21,209 \tHypothesis: die voorwerp van die tweede kategorie van komitees is om die uitvoer van persone buite die kommissie te verrig en te verseker dat gemeenskapsbetrokkenheid by die aktiwiteite van die kommissie geniet .\n",
            "2019-11-26 13:42:21,209 Example #3\n",
            "2019-11-26 13:42:21,209 \tSource:     * private acts by individuals\n",
            "2019-11-26 13:42:21,209 \tReference:  * indiwidue se private optrede\n",
            "2019-11-26 13:42:21,209 \tHypothesis: * private akte deur individue\n",
            "2019-11-26 13:42:21,209 Validation result (greedy) at epoch  97, step    42000: bleu:  50.55, loss: 25650.0742, ppl:   3.0021, duration: 48.9310s\n",
            "2019-11-26 13:42:37,346 Epoch  97 Step:    42100 Batch Loss:     0.730989 Tokens per Sec:    15465, Lr: 0.000300\n",
            "2019-11-26 13:42:53,345 Epoch  97 Step:    42200 Batch Loss:     0.697391 Tokens per Sec:    15646, Lr: 0.000300\n",
            "2019-11-26 13:43:09,412 Epoch  97 Step:    42300 Batch Loss:     0.723275 Tokens per Sec:    15356, Lr: 0.000300\n",
            "2019-11-26 13:43:16,901 Epoch  97: total training loss 296.88\n",
            "2019-11-26 13:43:16,901 EPOCH 98\n",
            "2019-11-26 13:43:25,529 Epoch  98 Step:    42400 Batch Loss:     0.581989 Tokens per Sec:    15189, Lr: 0.000300\n",
            "2019-11-26 13:43:41,195 Epoch  98 Step:    42500 Batch Loss:     0.941858 Tokens per Sec:    15275, Lr: 0.000300\n",
            "2019-11-26 13:43:57,262 Epoch  98 Step:    42600 Batch Loss:     0.701926 Tokens per Sec:    15269, Lr: 0.000300\n",
            "2019-11-26 13:44:13,271 Epoch  98 Step:    42700 Batch Loss:     0.465616 Tokens per Sec:    15611, Lr: 0.000300\n",
            "2019-11-26 13:44:26,759 Epoch  98: total training loss 296.85\n",
            "2019-11-26 13:44:26,759 EPOCH 99\n",
            "2019-11-26 13:44:29,105 Epoch  99 Step:    42800 Batch Loss:     0.589439 Tokens per Sec:    14413, Lr: 0.000300\n",
            "2019-11-26 13:44:45,198 Epoch  99 Step:    42900 Batch Loss:     0.503430 Tokens per Sec:    15244, Lr: 0.000300\n",
            "2019-11-26 13:45:01,427 Epoch  99 Step:    43000 Batch Loss:     0.546762 Tokens per Sec:    15396, Lr: 0.000300\n",
            "2019-11-26 13:45:17,476 Epoch  99 Step:    43100 Batch Loss:     0.561702 Tokens per Sec:    15548, Lr: 0.000300\n",
            "2019-11-26 13:45:33,518 Epoch  99 Step:    43200 Batch Loss:     0.968779 Tokens per Sec:    15407, Lr: 0.000300\n",
            "2019-11-26 13:45:36,676 Epoch  99: total training loss 292.35\n",
            "2019-11-26 13:45:36,676 EPOCH 100\n",
            "2019-11-26 13:45:49,484 Epoch 100 Step:    43300 Batch Loss:     0.821155 Tokens per Sec:    15152, Lr: 0.000300\n",
            "2019-11-26 13:46:05,568 Epoch 100 Step:    43400 Batch Loss:     0.401050 Tokens per Sec:    15539, Lr: 0.000300\n",
            "2019-11-26 13:46:21,506 Epoch 100 Step:    43500 Batch Loss:     0.500488 Tokens per Sec:    15254, Lr: 0.000300\n",
            "2019-11-26 13:46:37,440 Epoch 100 Step:    43600 Batch Loss:     0.538274 Tokens per Sec:    15333, Lr: 0.000300\n",
            "2019-11-26 13:46:46,523 Epoch 100: total training loss 291.58\n",
            "2019-11-26 13:46:46,524 Training ended after 100 epochs.\n",
            "2019-11-26 13:46:46,524 Best validation result (greedy) at step    40000:   2.99 ppl.\n",
            "2019-11-26 13:47:16,886  dev bleu:  48.65 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 13:47:16,887 Translations saved to: models/enaf_transformer/00040000.hyps.dev\n",
            "2019-11-26 13:48:27,509 test bleu:  19.56 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 13:48:27,510 Translations saved to: models/enaf_transformer/00040000.hyps.test\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "outputId": "4befa276-f2e6-41ba-b10e-dce5bc6991d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# Copy the created models from the notebook storage to google drive for persistant storage \n",
        "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"/content/drive/My Drive/masakhane/en-af-baseline/models/enaf_transformer/\""
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot create symbolic link '/content/drive/My Drive/masakhane/en-af-baseline/models/enaf_transformer/best.ckpt': Function not implemented\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "outputId": "8aed15df-c9f2-4732-d80d-c97f6cdb6eda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 370
        }
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 2000\tLoss: 74854.25781\tPPL: 24.73169\tbleu: 12.36405\tLR: 0.00030000\t*\n",
            "Steps: 4000\tLoss: 52469.36719\tPPL: 9.47559\tbleu: 26.50927\tLR: 0.00030000\t*\n",
            "Steps: 6000\tLoss: 42362.46094\tPPL: 6.14452\tbleu: 33.55846\tLR: 0.00030000\t*\n",
            "Steps: 8000\tLoss: 37101.53125\tPPL: 4.90418\tbleu: 37.38237\tLR: 0.00030000\t*\n",
            "Steps: 10000\tLoss: 34156.69531\tPPL: 4.32270\tbleu: 39.24070\tLR: 0.00030000\t*\n",
            "Steps: 12000\tLoss: 32103.47656\tPPL: 3.95857\tbleu: 40.75340\tLR: 0.00030000\t*\n",
            "Steps: 14000\tLoss: 30750.45508\tPPL: 3.73555\tbleu: 42.70344\tLR: 0.00030000\t*\n",
            "Steps: 16000\tLoss: 29685.38477\tPPL: 3.56887\tbleu: 43.53423\tLR: 0.00030000\t*\n",
            "Steps: 18000\tLoss: 28864.55078\tPPL: 3.44550\tbleu: 44.48371\tLR: 0.00030000\t*\n",
            "Steps: 20000\tLoss: 28165.52148\tPPL: 3.34381\tbleu: 45.28709\tLR: 0.00030000\t*\n",
            "Steps: 22000\tLoss: 27861.96484\tPPL: 3.30059\tbleu: 45.67252\tLR: 0.00030000\t*\n",
            "Steps: 24000\tLoss: 27393.80469\tPPL: 3.23503\tbleu: 47.06020\tLR: 0.00030000\t*\n",
            "Steps: 26000\tLoss: 26914.48047\tPPL: 3.16925\tbleu: 47.45031\tLR: 0.00030000\t*\n",
            "Steps: 28000\tLoss: 26760.47656\tPPL: 3.14840\tbleu: 46.99819\tLR: 0.00030000\t*\n",
            "Steps: 30000\tLoss: 26577.52539\tPPL: 3.12381\tbleu: 47.76965\tLR: 0.00030000\t*\n",
            "Steps: 32000\tLoss: 26218.08008\tPPL: 3.07606\tbleu: 49.17810\tLR: 0.00030000\t*\n",
            "Steps: 34000\tLoss: 26035.16602\tPPL: 3.05204\tbleu: 49.27485\tLR: 0.00030000\t*\n",
            "Steps: 36000\tLoss: 25899.09375\tPPL: 3.03429\tbleu: 49.66901\tLR: 0.00030000\t*\n",
            "Steps: 38000\tLoss: 25706.09180\tPPL: 3.00929\tbleu: 50.04462\tLR: 0.00030000\t*\n",
            "Steps: 40000\tLoss: 25581.30078\tPPL: 2.99324\tbleu: 49.61441\tLR: 0.00030000\t*\n",
            "Steps: 42000\tLoss: 25650.07422\tPPL: 3.00208\tbleu: 50.54650\tLR: 0.00030000\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "outputId": "204d1778-99db-48d2-8af6-89053a46af53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-11-26 13:49:10,919 Hello! This is Joey-NMT.\n",
            "2019-11-26 13:49:45,068  dev bleu:  48.65 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-11-26 13:50:55,979 test bleu:  19.56 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k9QbsG8rCJBX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}