en-nr/jw300-baseline/jw300_nr_starter_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "jw300_nr_starter_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "e9ec9df9-558a-4a0b-c1fd-cead0e0e6e51",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"nr\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "478895c4-0d83-44cb-fe90-2f07d60ba973",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-nr-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "1e009941-259e-4ec8-9bfd-4853d34c44b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 2.0MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "outputId": "ca1cce63-f219-4049-9bb6-ce5c5a34ee12",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        }
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-nr.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            " 940 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-nr.xml.gz\n",
            " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
            "   9 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/nr.zip\n",
            "\n",
            " 273 MB Total size\n",
            "./JW300_latest_xml_en-nr.xml.gz ... 100% of 940 KB\n",
            "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
            "./JW300_latest_xml_nr.zip ... 100% of 9 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n48GDRnP8y2G",
        "outputId": "1cfc86cf-c2b8-4c22-863c-3ce13c593e83",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-05-25 09:14:54--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.06s   \n",
            "\n",
            "2020-05-25 09:14:55 (4.67 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2020-05-25 09:14:57--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nr.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 203073 (198K) [text/plain]\n",
            "Saving to: ‘test.en-nr.en’\n",
            "\n",
            "test.en-nr.en       100%[===================>] 198.31K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2020-05-25 09:14:58 (4.36 MB/s) - ‘test.en-nr.en’ saved [203073/203073]\n",
            "\n",
            "--2020-05-25 09:15:04--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nr.nr\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 214288 (209K) [text/plain]\n",
            "Saving to: ‘test.en-nr.nr’\n",
            "\n",
            "test.en-nr.nr       100%[===================>] 209.27K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2020-05-25 09:15:04 (3.95 MB/s) - ‘test.en-nr.nr’ saved [214288/214288]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NqDG-CI28y2L",
        "outputId": "27a34eea-3da6-47df-f695-cdd96b66a825",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "b20d4dcf-e754-4400-ef59-3863072ced7f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 4805/103982 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>July 1 , 2010</td>\n",
              "      <td>Arhostosi 1 , 2010</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Do You Know God by Name ?</td>\n",
              "      <td>Umazi Kuhle na UZimu ?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>FROM OUR COVER</td>\n",
              "      <td>EZIKHAMBISANA NESIHLOKO ESINGAPHANDLE</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             source_sentence                        target_sentence\n",
              "0              July 1 , 2010                     Arhostosi 1 , 2010\n",
              "1  Do You Know God by Name ?                 Umazi Kuhle na UZimu ?\n",
              "2             FROM OUR COVER  EZIKHAMBISANA NESIHLOKO ESINGAPHANDLE"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "de0346f2-1937-423f-b485-cad958d5afc4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Z_1BwAApEtMk",
        "outputId": "19088ed8-f527-4d9b-b664-fc63bdcbb065",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "from os import cpu_count\n",
        "from functools import partial\n",
        "from multiprocessing import Pool\n",
        "\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/43/ff/74f23998ad2f93b945c0309f825be92e04e0348e062026998b5eefef4c33/fuzzywuzzy-0.18.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.18.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 1.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (46.3.0)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144790 sha256=5eb4cf12095d280b3add26009cbdea5ca779d1937901a45670b1535cb342e3a0\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5cCxB4ewIpMH",
        "colab_type": "code",
        "outputId": "f344d8dc-daf2-4526-888c-98d19bbd8629",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "start_time = time.time()\n",
        "### iterating over pandas dataframe rows is not recomended, let use multi processing to apply the function\n",
        "\n",
        "with Pool(cpu_count()-1) as pool:\n",
        "    scores = pool.map(partial(fuzzfilter, candidates=list(en_test_sents), pad=5), df_pp['source_sentence'])\n",
        "hours, rem = divmod(time.time() - start_time, 3600)\n",
        "minutes, seconds = divmod(rem, 60)\n",
        "print(\"done in {}h:{}min:{}seconds\".format(hours, minutes, seconds))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp = df_pp.assign(scores=scores)\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "done in 0.0h:36.0min:35.639527320861816seconds\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "outputId": "c04fef30-6f3c-4ddd-8de9-9fb50d028e8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 810
        }
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "Jehovah’s servants in the past set the pattern in their relationship with governments and officials .\n",
            "In that state , the insect resembles a berry , about the size and form of a pea , attached to the leaves and twigs of the kermes oak .\n",
            "( a ) What is the good news that we spread ?\n",
            "After all , “ a cheerful glance makes the heart rejoice . ”\n",
            "( b ) What questions can be helpful ?\n",
            "When a Loved One Is Terminally Ill , No .\n",
            "As they shared with him God’s promise that “ the lame one will climb up just as a stag does , ” the man responded with a wide smile . ​ — Isa .\n",
            "These men all had Greek names , which seems to indicate that the apostles wanted to ease any tensions over background that might have existed among the early Christians . ​ — Acts 6 : 2 - 6 .\n",
            "Religion Man - Made ?\n",
            "Do Not Let the Faults of Others Stumble You , June\n",
            "\n",
            "==> train.nr <==\n",
            "Iinceku zakaJehova zemandulo zisivulele umtlhala endleleni yokusebenzisana neenkhulu zakarhulumende .\n",
            "Ingogwana leyo neyinamaqanda , ifana nomomori ( berry ) , opheze ulingane begodu wakheke njengentanga yerhruni , enamathela emakarini nematakeni we - kermes oak .\n",
            "( a ) Ngiziphi iindaba ezimnandi esizirhatjhako ?\n",
            "“ Ubuso obuthokozileko buthabisa ihliziyo . ” ( IzA .\n",
            "( b ) Ngimiphi imibuzo engaba lisizo kithi ?\n",
            "U - Elias Hutter NamaBhayibhelakhe WesiHebheru , No .\n",
            "Bathi nabakhuluma nayo ngesithembiso sakaZimu sokuthi “ abarholopheleko bazakutjakadula njengehlangumataka , ” indoda leyo yabobotheka . ​ — Isa .\n",
            "Begodu woke amadoda akhethwako lawo bekanamagama wesiGirigi , lokhu kungenzeka ukuthi kwenzeka ngebanga lokuthi abapostoli bebafuna ukuqeda indaba yebandlululo le iphele nya . — IzE .\n",
            "Kuqakathekile Na Ukuba ngeweKolo ?\n",
            "Fumbelani Amatshwenyekwenu KuJehova , Dis .\n",
            "==> dev.en <==\n",
            "The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "Who was this Joseph of Arimathea ?\n",
            "Do you want to live in a world without Satan ?\n",
            "They are not suffering or in any kind of pain , for “ the dead know nothing at all . ”\n",
            "“ You are worthy , Jehovah our God , to receive the glory and the honor and the power , because you created all things . ” ​ — REV .\n",
            "You will also see that the Bible predicted such conditions for the period of time called “ the last days . ”\n",
            "Sophia : I never heard that part of the story before .\n",
            "Sílvia , a two - time cancer survivor , agrees . “ Having different friends drive me to another town daily for radiation was so relaxing and comforting !\n",
            "\n",
            "==> dev.nr <==\n",
            "Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "Kanti ngubani uJosefa we - Arimathiya ?\n",
            "Uyafuna na ukuphila ephasini elinganaSathana ?\n",
            "Abe abukho nobuncani ubuhlungu ababuzwako , nganingoba “ abazi litho . ”\n",
            "“ Ufanele wena [ “ Jehova , ” NW ] Zimethu , ukwamukela idumo , ubukhosi namandla ngombana nguwe owabumba koke begodu koke okukhona kwadalwa ngentando yakho . ” — ISAM .\n",
            "Godu uzokubona nokobana iBhayibhili labikezela ngobujamo obunjalo njengesikhathi esibizwa ngokobana ‘ mimihla yokuphela . ’\n",
            "Lindiwe : Ngiyathoma ukuyizwa indaba le .\n",
            "U - Sílvia , okhe waphathwa yikankere kabili uvumelana nalokho , uthi : “ Ukuba nabangani abahlukahlukeneko abangisa kelinye idorobho nengiya emtjhinini wokutjhisa amaseli wekankere [ radiation ] bekungiqabula begodu kungiduduza !\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "outputId": "a7378b60-ff55-4ad0-943b-a372a15cefcd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 28, done.\u001b[K\n",
            "remote: Counting objects: 100% (28/28), done.\u001b[K\n",
            "remote: Compressing objects: 100% (27/27), done.\u001b[K\n",
            "remote: Total 2423 (delta 9), reused 6 (delta 1), pack-reused 2395\u001b[K\n",
            "Receiving objects: 100% (2423/2423), 2.64 MiB | 2.48 MiB/s, done.\n",
            "Resolving deltas: 100% (1688/1688), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (7.0.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.18.4)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (46.3.0)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.5.0+cu101)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6e/9d/9846507837ca50ae20917f59d83b79246b8313bd19d4f5bf575ecb98132b/sacrebleu-1.4.9-py3-none-any.whl (60kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 1.7MB/s \n",
            "\u001b[?25hCollecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.10.1)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 6.9MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/37/6e/36419ec1bd2208e157dff7fc3e565b185394c0dc4901e9e2f983cb1d4b7f/pylint-2.5.2-py3-none-any.whl (324kB)\n",
            "\u001b[K     |████████████████████████████████| 327kB 20.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Collecting wrapt==1.11.1\n",
            "  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
            "Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.1)\n",
            "Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.10.0)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.3.3)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.6.3)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.29.0)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.34.2)\n",
            "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.4.1)\n",
            "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.41.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.23.0)\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/53/84/7b3146ec6378d28abc73ab484f09f47dfa008ad6f03f33d90a369f880e25/portalocker-1.7.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.2.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.8.1)\n",
            "Requirement already satisfied: pandas>=0.22.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.0.3)\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 6.9MB/s \n",
            "\u001b[?25hCollecting astroid<=2.5,>=2.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/46/c9/e9c2642dfb169590fb8bdb395f9329da042ee559c2ae7c1e612a3e5f40b4/astroid-2.4.1-py3-none-any.whl (214kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 22.9MB/s \n",
            "\u001b[?25hCollecting toml>=0.7.1\n",
            "  Downloading https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl\n",
            "Collecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.1)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.7.2)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.6.0.post3)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.2.2)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.0.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.9)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2020.4.5.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.22.0->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 23.7MB/s \n",
            "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 6.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.3.0)\n",
            "Requirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.2.8)\n",
            "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (4.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.6.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Building wheels for collected packages: joeynmt, pyyaml, wrapt\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=77165 sha256=52bcb5f17e336496836891392b4a8a70bdce6369cbc2d5abf41a2216c31d326a\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-rapd13ay/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44621 sha256=e9f18920f01d2557f16bcff32f39acc9e060c18281a0587b5f148f3a7abd7f11\n",
            "  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
            "  Building wheel for wrapt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wrapt: filename=wrapt-1.11.1-cp36-cp36m-linux_x86_64.whl size=67430 sha256=5f2ac61326aa6d4fa76935d2afc4c77c5ab400c9647dc1f21840b63cd47b7d75\n",
            "  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc\n",
            "Successfully built joeynmt pyyaml wrapt\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, isort, wrapt, typed-ast, lazy-object-proxy, astroid, toml, mccabe, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Found existing installation: wrapt 1.12.1\n",
            "    Uninstalling wrapt-1.12.1:\n",
            "      Successfully uninstalled wrapt-1.12.1\n",
            "Successfully installed astroid-2.4.1 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.7.0 pylint-2.5.2 pyyaml-5.3.1 sacrebleu-1.4.9 subword-nmt-0.3.7 toml-0.10.1 typed-ast-1.4.1 wrapt-1.11.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "outputId": "52432828-4677-414f-e263-3624a1cffb23",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        }
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 40000 -o bpe.codes.40000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.40000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.40000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE isiNdebele Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.40000  dev.en       test.bpe.nr     test.nr\t    train.en\n",
            "dev.bpe.en\t dev.nr       test.en\t      train.bpe.en  train.nr\n",
            "dev.bpe.nr\t test.bpe.en  test.en-any.en  train.bpe.nr  vocab.txt\n",
            "bpe.codes.40000  dev.en       test.bpe.nr     test.nr\t    train.en\n",
            "dev.bpe.en\t dev.nr       test.en\t      train.bpe.en  train.nr\n",
            "dev.bpe.nr\t test.bpe.en  test.en-any.en  train.bpe.nr\n",
            "BPE isiNdebele Sentences\n",
            "Lokho kwenza bona ngaz@@ iwe njengomuntu ong@@ akathembeki .\n",
            "Kwathi bona ngifunde iqiniso , akhenge ngis@@ avuma ukuraga nent@@ wel@@ eyo , nanyana umberego lo bewub@@ hadela kangaka .\n",
            "Ng@@ isibonelo esihle eb@@ as@@ any@@ aneni bami ababili , begodu ngikghona nokub@@ eregiswa ebandleni .\n",
            "Ebantwini abab@@ had@@ elisa umthelo nakilabo engib@@ ereg@@ isana nabo ngaz@@ iwa njengomuntu othembekileko . ”\n",
            "UR@@ ute wafudukela kwa - Israyeli lapho ebekazoku@@ kghona ukulotjha khona uZimu weqiniso .\n",
            "Combined BPE Vocab\n",
            "tirement\n",
            "Irel@@\n",
            "lier\n",
            "willing@@\n",
            "WITNES@@\n",
            "arrang@@\n",
            "ital@@\n",
            "IZWI\n",
            "build@@\n",
            "OKUBUZ@@\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlMitUHR8Qy-",
        "outputId": "4064b14a-fde0-4a37-c381-7d9dd4198571",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.40000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\""
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.40000  dev.en       test.bpe.nr     test.nr\t    train.en\n",
            "dev.bpe.en\t dev.nr       test.en\t      train.bpe.en  train.nr\n",
            "dev.bpe.nr\t test.bpe.en  test.en-any.en  train.bpe.nr\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iprRFmA1e_cn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#THIS WAS NEVER SAVED TO DRIVE.\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py \"/content/drive/My Drive/masakhane/en-nr-baseline/train.bpe.$src\" \"/content/drive/My Drive/masakhane/en-nr-baseline/train.bpe.$tgt\" --output_path \"/content/drive/My Drive/masakhane/en-nr-baseline/vocab.txt\"\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"data/{name}/train.bpe\"\n",
        "    dev:   \"data/{name}/dev.bpe\"\n",
        "    test:  \"data/{name}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"data/{name}/vocab.txt\"\n",
        "    trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 30                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "outputId": "949abe85-da56-45b6-9ca5-ab191e007c6f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-05-25 10:07:25,863 Hello! This is Joey-NMT.\n",
            "2020-05-25 10:07:25.998377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-05-25 10:07:27,575 Total params: 21083392\n",
            "2020-05-25 10:07:27,576 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2020-05-25 10:07:41,653 cfg.name                           : ennr_transformer\n",
            "2020-05-25 10:07:41,653 cfg.data.src                       : en\n",
            "2020-05-25 10:07:41,654 cfg.data.trg                       : nr\n",
            "2020-05-25 10:07:41,654 cfg.data.train                     : data/ennr/train.bpe\n",
            "2020-05-25 10:07:41,654 cfg.data.dev                       : data/ennr/dev.bpe\n",
            "2020-05-25 10:07:41,654 cfg.data.test                      : data/ennr/test.bpe\n",
            "2020-05-25 10:07:41,654 cfg.data.level                     : bpe\n",
            "2020-05-25 10:07:41,654 cfg.data.lowercase                 : False\n",
            "2020-05-25 10:07:41,654 cfg.data.max_sent_length           : 100\n",
            "2020-05-25 10:07:41,654 cfg.data.src_vocab                 : data/ennr/vocab.txt\n",
            "2020-05-25 10:07:41,654 cfg.data.trg_vocab                 : data/ennr/vocab.txt\n",
            "2020-05-25 10:07:41,654 cfg.testing.beam_size              : 5\n",
            "2020-05-25 10:07:41,655 cfg.testing.alpha                  : 1.0\n",
            "2020-05-25 10:07:41,655 cfg.training.random_seed           : 42\n",
            "2020-05-25 10:07:41,655 cfg.training.optimizer             : adam\n",
            "2020-05-25 10:07:41,655 cfg.training.normalization         : tokens\n",
            "2020-05-25 10:07:41,655 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2020-05-25 10:07:41,655 cfg.training.scheduling            : plateau\n",
            "2020-05-25 10:07:41,655 cfg.training.patience              : 5\n",
            "2020-05-25 10:07:41,655 cfg.training.learning_rate_factor  : 0.5\n",
            "2020-05-25 10:07:41,655 cfg.training.learning_rate_warmup  : 1000\n",
            "2020-05-25 10:07:41,656 cfg.training.decrease_factor       : 0.7\n",
            "2020-05-25 10:07:41,656 cfg.training.loss                  : crossentropy\n",
            "2020-05-25 10:07:41,656 cfg.training.learning_rate         : 0.0003\n",
            "2020-05-25 10:07:41,656 cfg.training.learning_rate_min     : 1e-08\n",
            "2020-05-25 10:07:41,656 cfg.training.weight_decay          : 0.0\n",
            "2020-05-25 10:07:41,656 cfg.training.label_smoothing       : 0.1\n",
            "2020-05-25 10:07:41,656 cfg.training.batch_size            : 4096\n",
            "2020-05-25 10:07:41,656 cfg.training.batch_type            : token\n",
            "2020-05-25 10:07:41,656 cfg.training.eval_batch_size       : 3600\n",
            "2020-05-25 10:07:41,656 cfg.training.eval_batch_type       : token\n",
            "2020-05-25 10:07:41,657 cfg.training.batch_multiplier      : 1\n",
            "2020-05-25 10:07:41,657 cfg.training.early_stopping_metric : ppl\n",
            "2020-05-25 10:07:41,657 cfg.training.epochs                : 30\n",
            "2020-05-25 10:07:41,657 cfg.training.validation_freq       : 1000\n",
            "2020-05-25 10:07:41,657 cfg.training.logging_freq          : 100\n",
            "2020-05-25 10:07:41,657 cfg.training.eval_metric           : bleu\n",
            "2020-05-25 10:07:41,657 cfg.training.model_dir             : models/ennr_transformer\n",
            "2020-05-25 10:07:41,657 cfg.training.overwrite             : False\n",
            "2020-05-25 10:07:41,657 cfg.training.shuffle               : True\n",
            "2020-05-25 10:07:41,658 cfg.training.use_cuda              : True\n",
            "2020-05-25 10:07:41,658 cfg.training.max_output_length     : 100\n",
            "2020-05-25 10:07:41,658 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2020-05-25 10:07:41,658 cfg.training.keep_last_ckpts       : 3\n",
            "2020-05-25 10:07:41,658 cfg.model.initializer              : xavier\n",
            "2020-05-25 10:07:41,658 cfg.model.bias_initializer         : zeros\n",
            "2020-05-25 10:07:41,658 cfg.model.init_gain                : 1.0\n",
            "2020-05-25 10:07:41,658 cfg.model.embed_initializer        : xavier\n",
            "2020-05-25 10:07:41,658 cfg.model.embed_init_gain          : 1.0\n",
            "2020-05-25 10:07:41,658 cfg.model.tied_embeddings          : True\n",
            "2020-05-25 10:07:41,659 cfg.model.tied_softmax             : True\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.type             : transformer\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.num_layers       : 6\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.num_heads        : 4\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.embeddings.scale : True\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.hidden_size      : 256\n",
            "2020-05-25 10:07:41,659 cfg.model.encoder.ff_size          : 1024\n",
            "2020-05-25 10:07:41,660 cfg.model.encoder.dropout          : 0.3\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.type             : transformer\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.num_layers       : 6\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.num_heads        : 4\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.embeddings.scale : True\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.hidden_size      : 256\n",
            "2020-05-25 10:07:41,660 cfg.model.decoder.ff_size          : 1024\n",
            "2020-05-25 10:07:41,661 cfg.model.decoder.dropout          : 0.3\n",
            "2020-05-25 10:07:41,661 Data set sizes: \n",
            "\ttrain 90654,\n",
            "\tvalid 1000,\n",
            "\ttest 2672\n",
            "2020-05-25 10:07:41,661 First training example:\n",
            "\t[SRC] Jehovah’s servants in the past set the pattern in their relationship with governments and officials .\n",
            "\t[TRG] Iinceku zakaJehova zem@@ andulo zis@@ iv@@ ulele umtlhala endleleni yoku@@ sebenzisana neenkhulu zakarhulumende .\n",
            "2020-05-25 10:07:41,661 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) of (9) :\n",
            "2020-05-25 10:07:41,662 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) of (9) :\n",
            "2020-05-25 10:07:41,662 Number of Src words (types): 39153\n",
            "2020-05-25 10:07:41,662 Number of Trg words (types): 39153\n",
            "2020-05-25 10:07:41,662 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=39153),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=39153))\n",
            "2020-05-25 10:07:41,693 EPOCH 1\n",
            "/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:\n",
            "\tnonzero(Tensor input, *, Tensor out)\n",
            "Consider using one of the following signatures instead:\n",
            "\tnonzero(Tensor input, *, bool as_tuple)\n",
            "2020-05-25 10:08:32,411 Epoch   1 Step:      100 Batch Loss:     7.211356 Tokens per Sec:     3953, Lr: 0.000300\n",
            "2020-05-25 10:09:22,676 Epoch   1 Step:      200 Batch Loss:     6.965822 Tokens per Sec:     4050, Lr: 0.000300\n",
            "2020-05-25 10:10:13,677 Epoch   1 Step:      300 Batch Loss:     6.630541 Tokens per Sec:     3998, Lr: 0.000300\n",
            "2020-05-25 10:11:04,519 Epoch   1 Step:      400 Batch Loss:     6.411305 Tokens per Sec:     3933, Lr: 0.000300\n",
            "2020-05-25 10:11:55,309 Epoch   1 Step:      500 Batch Loss:     6.247663 Tokens per Sec:     3897, Lr: 0.000300\n",
            "2020-05-25 10:12:45,993 Epoch   1 Step:      600 Batch Loss:     6.483799 Tokens per Sec:     3952, Lr: 0.000300\n",
            "2020-05-25 10:13:37,553 Epoch   1 Step:      700 Batch Loss:     6.492374 Tokens per Sec:     3933, Lr: 0.000300\n",
            "2020-05-25 10:14:10,419 Epoch   1: total training loss 5081.31\n",
            "2020-05-25 10:14:10,419 EPOCH 2\n",
            "2020-05-25 10:14:28,741 Epoch   2 Step:      800 Batch Loss:     6.017582 Tokens per Sec:     3941, Lr: 0.000300\n",
            "2020-05-25 10:15:19,563 Epoch   2 Step:      900 Batch Loss:     5.772392 Tokens per Sec:     3955, Lr: 0.000300\n",
            "2020-05-25 10:16:11,347 Epoch   2 Step:     1000 Batch Loss:     5.954979 Tokens per Sec:     3935, Lr: 0.000300\n",
            "2020-05-25 10:18:33,414 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 10:18:33,414 Saving new checkpoint.\n",
            "2020-05-25 10:18:33,880 Example #0\n",
            "2020-05-25 10:18:33,880 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 10:18:33,880 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 10:18:33,880 \tHypothesis: Kodwana , uJehova uJehova , uJehova uJehova , uJehova uJehova uJehova , uJehova uJehova .\n",
            "2020-05-25 10:18:33,880 Example #1\n",
            "2020-05-25 10:18:33,881 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 10:18:33,881 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 10:18:33,881 \tHypothesis: Nanyana kunjalo , “ “ “ “ “ “ ‘ kobana kobana uJehova uJehova , “ “ “ “ “ “ ‘ kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana kobana\n",
            "2020-05-25 10:18:33,881 Example #2\n",
            "2020-05-25 10:18:33,881 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 10:18:33,881 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 10:18:33,881 \tHypothesis: Kodwana , uJehova uJehova , uJehova uJehova , uJehova kufuze uJehova uJehova , uJehova kufuze uJehova uJehova uJehova , uJehova uJehova kufuze uJehova uJehova uJehova uJehova uJehova uJehova , uJehova uJehova , uJehova uJehova uJehova uJehova uJehova , uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova , uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova kufuze uJehova kufuze uJehova kufuze uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova uJehova kufuze uJehova kufuze uJehova uJehova uJehova\n",
            "2020-05-25 10:18:33,881 Example #3\n",
            "2020-05-25 10:18:33,882 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 10:18:33,882 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 10:18:33,882 \tHypothesis: Kubayini kufuze kubayini uJehova ?\n",
            "2020-05-25 10:18:33,882 Validation result (greedy) at epoch   2, step     1000: bleu:   0.12, loss: 99971.8828, ppl: 336.5500, duration: 142.5343s\n",
            "2020-05-25 10:19:25,725 Epoch   2 Step:     1100 Batch Loss:     5.877172 Tokens per Sec:     4030, Lr: 0.000300\n",
            "2020-05-25 10:20:16,523 Epoch   2 Step:     1200 Batch Loss:     6.005499 Tokens per Sec:     3870, Lr: 0.000300\n",
            "2020-05-25 10:21:07,776 Epoch   2 Step:     1300 Batch Loss:     5.445710 Tokens per Sec:     3977, Lr: 0.000300\n",
            "2020-05-25 10:21:59,053 Epoch   2 Step:     1400 Batch Loss:     5.715794 Tokens per Sec:     3909, Lr: 0.000300\n",
            "2020-05-25 10:22:50,096 Epoch   2 Step:     1500 Batch Loss:     5.804766 Tokens per Sec:     3935, Lr: 0.000300\n",
            "2020-05-25 10:23:02,608 Epoch   2: total training loss 4365.97\n",
            "2020-05-25 10:23:02,608 EPOCH 3\n",
            "2020-05-25 10:23:41,646 Epoch   3 Step:     1600 Batch Loss:     5.540798 Tokens per Sec:     3997, Lr: 0.000300\n",
            "2020-05-25 10:24:32,591 Epoch   3 Step:     1700 Batch Loss:     5.530884 Tokens per Sec:     3981, Lr: 0.000300\n",
            "2020-05-25 10:25:24,414 Epoch   3 Step:     1800 Batch Loss:     5.342935 Tokens per Sec:     3931, Lr: 0.000300\n",
            "2020-05-25 10:26:15,760 Epoch   3 Step:     1900 Batch Loss:     5.219332 Tokens per Sec:     3987, Lr: 0.000300\n",
            "2020-05-25 10:27:07,122 Epoch   3 Step:     2000 Batch Loss:     5.275292 Tokens per Sec:     3909, Lr: 0.000300\n",
            "2020-05-25 10:29:17,487 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 10:29:17,487 Saving new checkpoint.\n",
            "2020-05-25 10:29:17,939 Example #0\n",
            "2020-05-25 10:29:17,939 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 10:29:17,939 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 10:29:17,939 \tHypothesis: U - Israyeli , u - Israyeli , kodwana akhenge amkhala .\n",
            "2020-05-25 10:29:17,939 Example #1\n",
            "2020-05-25 10:29:17,940 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 10:29:17,940 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 10:29:17,940 \tHypothesis: UJesu wathi : “ Ngazi bona uJesu wathi : “ Ngazi bona uJesu , kodwana akhenge amkhala . ”\n",
            "2020-05-25 10:29:17,940 Example #2\n",
            "2020-05-25 10:29:17,940 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 10:29:17,940 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 10:29:17,940 \tHypothesis: Ngemva kwalokho , uJesu wathi : “ Ngazi , kodwana akhenge akhenge akhenge akhenge akhenge akhenge akhenge amkhala .\n",
            "2020-05-25 10:29:17,940 Example #3\n",
            "2020-05-25 10:29:17,941 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 10:29:17,941 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 10:29:17,941 \tHypothesis: Kghani uJesu - ke - ke - ke - ke , kubayini ?\n",
            "2020-05-25 10:29:17,941 Validation result (greedy) at epoch   3, step     2000: bleu:   1.25, loss: 87661.0547, ppl: 164.3846, duration: 130.8184s\n",
            "2020-05-25 10:30:08,993 Epoch   3 Step:     2100 Batch Loss:     5.213541 Tokens per Sec:     3923, Lr: 0.000300\n",
            "2020-05-25 10:31:00,068 Epoch   3 Step:     2200 Batch Loss:     5.015653 Tokens per Sec:     3857, Lr: 0.000300\n",
            "2020-05-25 10:31:43,287 Epoch   3: total training loss 3962.32\n",
            "2020-05-25 10:31:43,287 EPOCH 4\n",
            "2020-05-25 10:31:51,852 Epoch   4 Step:     2300 Batch Loss:     4.895185 Tokens per Sec:     3812, Lr: 0.000300\n",
            "2020-05-25 10:32:42,836 Epoch   4 Step:     2400 Batch Loss:     4.647756 Tokens per Sec:     3906, Lr: 0.000300\n",
            "2020-05-25 10:33:33,253 Epoch   4 Step:     2500 Batch Loss:     5.308408 Tokens per Sec:     3881, Lr: 0.000300\n",
            "2020-05-25 10:34:24,714 Epoch   4 Step:     2600 Batch Loss:     4.777163 Tokens per Sec:     3953, Lr: 0.000300\n",
            "2020-05-25 10:35:15,858 Epoch   4 Step:     2700 Batch Loss:     3.514590 Tokens per Sec:     3977, Lr: 0.000300\n",
            "2020-05-25 10:36:06,932 Epoch   4 Step:     2800 Batch Loss:     4.947702 Tokens per Sec:     3921, Lr: 0.000300\n",
            "2020-05-25 10:36:58,432 Epoch   4 Step:     2900 Batch Loss:     4.938779 Tokens per Sec:     3998, Lr: 0.000300\n",
            "2020-05-25 10:37:49,106 Epoch   4 Step:     3000 Batch Loss:     4.795860 Tokens per Sec:     3897, Lr: 0.000300\n",
            "2020-05-25 10:39:12,653 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 10:39:12,653 Saving new checkpoint.\n",
            "2020-05-25 10:39:13,098 Example #0\n",
            "2020-05-25 10:39:13,098 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 10:39:13,098 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 10:39:13,098 \tHypothesis: Ngafika , u - Israyeli , ama - Israyeli bekaba bakhe .\n",
            "2020-05-25 10:39:13,098 Example #1\n",
            "2020-05-25 10:39:13,099 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 10:39:13,099 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 10:39:13,099 \tHypothesis: U - Adamu no - Adamu no - Adamu , wathi : “ Ngaba - Adamu , kodwana mina mina , kodwana mina mina . . . . . . . . . . . . .\n",
            "2020-05-25 10:39:13,099 Example #2\n",
            "2020-05-25 10:39:13,099 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 10:39:13,099 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 10:39:13,099 \tHypothesis: Wathi , bekangatjho bona umuntu angamtjela bona abe nesibindi , kodwana yena wathi : “ Ngambile .\n",
            "2020-05-25 10:39:13,099 Example #3\n",
            "2020-05-25 10:39:13,100 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 10:39:13,100 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 10:39:13,100 \tHypothesis: Kghani u - kghani u - Arhabi ?\n",
            "2020-05-25 10:39:13,100 Validation result (greedy) at epoch   4, step     3000: bleu:   3.11, loss: 78894.1953, ppl:  98.6861, duration: 83.9940s\n",
            "2020-05-25 10:39:37,331 Epoch   4: total training loss 3662.26\n",
            "2020-05-25 10:39:37,332 EPOCH 5\n",
            "2020-05-25 10:40:04,853 Epoch   5 Step:     3100 Batch Loss:     3.724206 Tokens per Sec:     4015, Lr: 0.000300\n",
            "2020-05-25 10:40:56,306 Epoch   5 Step:     3200 Batch Loss:     4.079641 Tokens per Sec:     3934, Lr: 0.000300\n",
            "2020-05-25 10:41:47,538 Epoch   5 Step:     3300 Batch Loss:     4.424605 Tokens per Sec:     3956, Lr: 0.000300\n",
            "2020-05-25 10:42:38,508 Epoch   5 Step:     3400 Batch Loss:     4.620626 Tokens per Sec:     3898, Lr: 0.000300\n",
            "2020-05-25 10:43:29,679 Epoch   5 Step:     3500 Batch Loss:     4.744503 Tokens per Sec:     4000, Lr: 0.000300\n",
            "2020-05-25 10:44:19,948 Epoch   5 Step:     3600 Batch Loss:     4.547628 Tokens per Sec:     3908, Lr: 0.000300\n",
            "2020-05-25 10:45:11,375 Epoch   5 Step:     3700 Batch Loss:     4.247553 Tokens per Sec:     3953, Lr: 0.000300\n",
            "2020-05-25 10:46:02,631 Epoch   5 Step:     3800 Batch Loss:     3.881932 Tokens per Sec:     3918, Lr: 0.000300\n",
            "2020-05-25 10:46:06,884 Epoch   5: total training loss 3392.48\n",
            "2020-05-25 10:46:06,884 EPOCH 6\n",
            "2020-05-25 10:46:53,841 Epoch   6 Step:     3900 Batch Loss:     2.895326 Tokens per Sec:     3920, Lr: 0.000300\n",
            "2020-05-25 10:47:44,850 Epoch   6 Step:     4000 Batch Loss:     3.987758 Tokens per Sec:     3973, Lr: 0.000300\n",
            "2020-05-25 10:48:53,816 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 10:48:53,816 Saving new checkpoint.\n",
            "2020-05-25 10:48:54,294 Example #0\n",
            "2020-05-25 10:48:54,294 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 10:48:54,295 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 10:48:54,295 \tHypothesis: UMesiya bekaphila hlangana nabantu bakhe , begodu bekukwosingazimu .\n",
            "2020-05-25 10:48:54,295 Example #1\n",
            "2020-05-25 10:48:54,295 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 10:48:54,295 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 10:48:54,295 \tHypothesis: U - Abrahama akhenge abe nesibindi , kodwana wathi : “ Mina , kodwana mina mina mina , kodwana mina mina mina , mina ! . . . . . . . . . .\n",
            "2020-05-25 10:48:54,295 Example #2\n",
            "2020-05-25 10:48:54,296 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 10:48:54,296 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 10:48:54,296 \tHypothesis: Wathi , wathi : “ Nanyana kunjalo , wenza njalo , ngombana umuntu angabawa bona abe nesibindi .\n",
            "2020-05-25 10:48:54,296 Example #3\n",
            "2020-05-25 10:48:54,296 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 10:48:54,296 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 10:48:54,296 \tHypothesis: Ngubani u - New World Translation ?\n",
            "2020-05-25 10:48:54,296 Validation result (greedy) at epoch   6, step     4000: bleu:   5.35, loss: 73069.3438, ppl:  70.3101, duration: 69.4459s\n",
            "2020-05-25 10:49:45,202 Epoch   6 Step:     4100 Batch Loss:     4.568571 Tokens per Sec:     3900, Lr: 0.000300\n",
            "2020-05-25 10:50:36,533 Epoch   6 Step:     4200 Batch Loss:     4.330948 Tokens per Sec:     4027, Lr: 0.000300\n",
            "2020-05-25 10:51:27,308 Epoch   6 Step:     4300 Batch Loss:     2.994790 Tokens per Sec:     3889, Lr: 0.000300\n",
            "2020-05-25 10:52:18,530 Epoch   6 Step:     4400 Batch Loss:     4.560994 Tokens per Sec:     4053, Lr: 0.000300\n",
            "2020-05-25 10:53:09,805 Epoch   6 Step:     4500 Batch Loss:     3.958454 Tokens per Sec:     3900, Lr: 0.000300\n",
            "2020-05-25 10:53:45,372 Epoch   6: total training loss 3186.30\n",
            "2020-05-25 10:53:45,373 EPOCH 7\n",
            "2020-05-25 10:54:00,658 Epoch   7 Step:     4600 Batch Loss:     4.307690 Tokens per Sec:     3932, Lr: 0.000300\n",
            "2020-05-25 10:54:51,731 Epoch   7 Step:     4700 Batch Loss:     2.707994 Tokens per Sec:     3938, Lr: 0.000300\n",
            "2020-05-25 10:55:42,721 Epoch   7 Step:     4800 Batch Loss:     4.294012 Tokens per Sec:     3925, Lr: 0.000300\n",
            "2020-05-25 10:56:33,535 Epoch   7 Step:     4900 Batch Loss:     4.175139 Tokens per Sec:     3928, Lr: 0.000300\n",
            "2020-05-25 10:57:25,213 Epoch   7 Step:     5000 Batch Loss:     2.907524 Tokens per Sec:     3996, Lr: 0.000300\n",
            "2020-05-25 10:58:21,095 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 10:58:21,095 Saving new checkpoint.\n",
            "2020-05-25 10:58:21,575 Example #0\n",
            "2020-05-25 10:58:21,575 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 10:58:21,575 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 10:58:21,575 \tHypothesis: Umpristi lo waba yingcenye yabantu , ama - Israyeli amanengi amanengi .\n",
            "2020-05-25 10:58:21,576 Example #1\n",
            "2020-05-25 10:58:21,576 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 10:58:21,576 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 10:58:21,576 \tHypothesis: U - Esewu bekangalemuka bona umuntu angakwenza umuntu ongamenza angatjho ukuthi : “ Mina mina mina , mina mina mina mina mina mina , mina mina mina mina !\n",
            "2020-05-25 10:58:21,576 Example #2\n",
            "2020-05-25 10:58:21,576 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 10:58:21,576 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 10:58:21,577 \tHypothesis: Nekafunda kuhle kuhle , akhenge angakwazi ukwenza njalo , kodwana umfazi lo watjela umfazi lo bona uMariya .\n",
            "2020-05-25 10:58:21,577 Example #3\n",
            "2020-05-25 10:58:21,577 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 10:58:21,577 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 10:58:21,577 \tHypothesis: Ngubani eyasiza u - ofisi legatja ?\n",
            "2020-05-25 10:58:21,577 Validation result (greedy) at epoch   7, step     5000: bleu:   8.09, loss: 68052.5703, ppl:  52.5056, duration: 56.3642s\n",
            "2020-05-25 10:59:12,424 Epoch   7 Step:     5100 Batch Loss:     4.321835 Tokens per Sec:     3944, Lr: 0.000300\n",
            "2020-05-25 11:00:03,672 Epoch   7 Step:     5200 Batch Loss:     4.078584 Tokens per Sec:     3970, Lr: 0.000300\n",
            "2020-05-25 11:00:54,677 Epoch   7 Step:     5300 Batch Loss:     4.178669 Tokens per Sec:     3853, Lr: 0.000300\n",
            "2020-05-25 11:01:11,940 Epoch   7: total training loss 3015.61\n",
            "2020-05-25 11:01:11,941 EPOCH 8\n",
            "2020-05-25 11:01:45,973 Epoch   8 Step:     5400 Batch Loss:     3.566381 Tokens per Sec:     3974, Lr: 0.000300\n",
            "2020-05-25 11:02:36,925 Epoch   8 Step:     5500 Batch Loss:     3.704456 Tokens per Sec:     3916, Lr: 0.000300\n",
            "2020-05-25 11:03:28,303 Epoch   8 Step:     5600 Batch Loss:     3.832468 Tokens per Sec:     4012, Lr: 0.000300\n",
            "2020-05-25 11:04:19,102 Epoch   8 Step:     5700 Batch Loss:     2.345056 Tokens per Sec:     3908, Lr: 0.000300\n",
            "2020-05-25 11:05:10,132 Epoch   8 Step:     5800 Batch Loss:     3.776027 Tokens per Sec:     4035, Lr: 0.000300\n",
            "2020-05-25 11:06:01,448 Epoch   8 Step:     5900 Batch Loss:     2.693149 Tokens per Sec:     3957, Lr: 0.000300\n",
            "2020-05-25 11:06:52,559 Epoch   8 Step:     6000 Batch Loss:     4.402835 Tokens per Sec:     3890, Lr: 0.000300\n",
            "2020-05-25 11:07:39,576 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:07:39,576 Saving new checkpoint.\n",
            "2020-05-25 11:07:40,066 Example #0\n",
            "2020-05-25 11:07:40,066 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:07:40,066 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:07:40,066 \tHypothesis: Ikosi le beyifanekisela amaJuda , ama - Israyeli bona abe mumuntu oyedwa .\n",
            "2020-05-25 11:07:40,066 Example #1\n",
            "2020-05-25 11:07:40,066 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:07:40,067 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:07:40,067 \tHypothesis: U - Esewu bekangalemuka bona kubayini lokho , watjela u - Isaka wathi : “ Mina mina mina mina mina , mina mina mina mina mina mina mina mina mina .\n",
            "2020-05-25 11:07:40,067 Example #2\n",
            "2020-05-25 11:07:40,067 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:07:40,067 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:07:40,067 \tHypothesis: Nekafunda naye , bekafuna bona angakhulumi naye , ngitjho nanyana ngubani owatjela uJesu bona abe yikosi .\n",
            "2020-05-25 11:07:40,068 Example #3\n",
            "2020-05-25 11:07:40,068 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:07:40,068 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:07:40,068 \tHypothesis: Ngubani u - Arimathiya ?\n",
            "2020-05-25 11:07:40,068 Validation result (greedy) at epoch   8, step     6000: bleu:  10.06, loss: 64672.6172, ppl:  43.1289, duration: 47.5090s\n",
            "2020-05-25 11:08:28,026 Epoch   8: total training loss 2842.54\n",
            "2020-05-25 11:08:28,026 EPOCH 9\n",
            "2020-05-25 11:08:31,708 Epoch   9 Step:     6100 Batch Loss:     2.596967 Tokens per Sec:     3940, Lr: 0.000300\n",
            "2020-05-25 11:09:23,004 Epoch   9 Step:     6200 Batch Loss:     3.824742 Tokens per Sec:     3903, Lr: 0.000300\n",
            "2020-05-25 11:10:14,675 Epoch   9 Step:     6300 Batch Loss:     3.485603 Tokens per Sec:     4009, Lr: 0.000300\n",
            "2020-05-25 11:11:06,065 Epoch   9 Step:     6400 Batch Loss:     3.679117 Tokens per Sec:     3988, Lr: 0.000300\n",
            "2020-05-25 11:11:57,414 Epoch   9 Step:     6500 Batch Loss:     3.425065 Tokens per Sec:     3977, Lr: 0.000300\n",
            "2020-05-25 11:12:48,187 Epoch   9 Step:     6600 Batch Loss:     3.333281 Tokens per Sec:     3857, Lr: 0.000300\n",
            "2020-05-25 11:13:38,756 Epoch   9 Step:     6700 Batch Loss:     3.282903 Tokens per Sec:     3966, Lr: 0.000300\n",
            "2020-05-25 11:14:29,194 Epoch   9 Step:     6800 Batch Loss:     2.681944 Tokens per Sec:     3893, Lr: 0.000300\n",
            "2020-05-25 11:14:57,881 Epoch   9: total training loss 2722.73\n",
            "2020-05-25 11:14:57,882 EPOCH 10\n",
            "2020-05-25 11:15:20,489 Epoch  10 Step:     6900 Batch Loss:     2.158492 Tokens per Sec:     3901, Lr: 0.000300\n",
            "2020-05-25 11:16:11,466 Epoch  10 Step:     7000 Batch Loss:     3.532651 Tokens per Sec:     3979, Lr: 0.000300\n",
            "2020-05-25 11:17:17,326 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:17:17,327 Saving new checkpoint.\n",
            "2020-05-25 11:17:17,753 Example #0\n",
            "2020-05-25 11:17:17,754 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:17:17,754 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:17:17,754 \tHypothesis: Ikosi yakhiwa babantu bakaZimu , abantu abanengi batjala ama - Israyeli bona benze okufanako .\n",
            "2020-05-25 11:17:17,754 Example #1\n",
            "2020-05-25 11:17:17,755 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:17:17,755 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:17:17,755 \tHypothesis: U - Esewu nekavumela bona kubayini lokho akuqala , u - Isaka wathi : “ Ngithanda , mina mina mina mina mina mina mina mina , mina mina mina mina mina mina mina ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
            "2020-05-25 11:17:17,755 Example #2\n",
            "2020-05-25 11:17:17,755 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:17:17,755 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:17:17,755 \tHypothesis: Nekafunda ngaye , bekafuna ukumlinga ukumlalela , ngitjho nanyana ngubani owatjela bona uJesu bekamtjela bona uMariya .\n",
            "2020-05-25 11:17:17,755 Example #3\n",
            "2020-05-25 11:17:17,756 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:17:17,756 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:17:17,756 \tHypothesis: Ngubani Okwenzeka ngo - Asiriya ?\n",
            "2020-05-25 11:17:17,756 Validation result (greedy) at epoch  10, step     7000: bleu:  11.18, loss: 61825.8711, ppl:  36.5435, duration: 66.2898s\n",
            "2020-05-25 11:18:08,947 Epoch  10 Step:     7100 Batch Loss:     3.364533 Tokens per Sec:     3952, Lr: 0.000300\n",
            "2020-05-25 11:19:00,120 Epoch  10 Step:     7200 Batch Loss:     3.256028 Tokens per Sec:     3933, Lr: 0.000300\n",
            "2020-05-25 11:19:51,621 Epoch  10 Step:     7300 Batch Loss:     3.800498 Tokens per Sec:     3981, Lr: 0.000300\n",
            "2020-05-25 11:20:42,851 Epoch  10 Step:     7400 Batch Loss:     2.939037 Tokens per Sec:     3962, Lr: 0.000300\n",
            "2020-05-25 11:21:34,109 Epoch  10 Step:     7500 Batch Loss:     2.867895 Tokens per Sec:     3926, Lr: 0.000300\n",
            "2020-05-25 11:22:25,440 Epoch  10 Step:     7600 Batch Loss:     3.335697 Tokens per Sec:     3992, Lr: 0.000300\n",
            "2020-05-25 11:22:32,887 Epoch  10: total training loss 2598.48\n",
            "2020-05-25 11:22:32,887 EPOCH 11\n",
            "2020-05-25 11:23:16,378 Epoch  11 Step:     7700 Batch Loss:     3.232729 Tokens per Sec:     3924, Lr: 0.000300\n",
            "2020-05-25 11:24:07,559 Epoch  11 Step:     7800 Batch Loss:     3.767534 Tokens per Sec:     3898, Lr: 0.000300\n",
            "2020-05-25 11:24:59,047 Epoch  11 Step:     7900 Batch Loss:     3.215782 Tokens per Sec:     3977, Lr: 0.000300\n",
            "2020-05-25 11:25:50,132 Epoch  11 Step:     8000 Batch Loss:     2.819639 Tokens per Sec:     3903, Lr: 0.000300\n",
            "2020-05-25 11:26:33,635 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:26:33,635 Saving new checkpoint.\n",
            "2020-05-25 11:26:34,228 Example #0\n",
            "2020-05-25 11:26:34,228 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:26:34,228 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:26:34,228 \tHypothesis: Ikosi yakhiwa ziinceku zaka - Israyeli , abanengi abanengi baba makhosi wakwa - Israyeli .\n",
            "2020-05-25 11:26:34,229 Example #1\n",
            "2020-05-25 11:26:34,229 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:26:34,229 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:26:34,229 \tHypothesis: U - Esewu nekavumela bona kubayini u - Esewu nekakhetha ukungafaneleki , wathi : “ Ningesabi , ngitjho nanyana ngingami ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
            "2020-05-25 11:26:34,229 Example #2\n",
            "2020-05-25 11:26:34,229 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:26:34,230 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:26:34,230 \tHypothesis: Nekafunda ngaye , bekafuna ukumzwisa ubuhlungu , bekafuna ukumlalela , ngitjho nanyana ngubani loyo ngaphambi kobana uJesu wathini .\n",
            "2020-05-25 11:26:34,230 Example #3\n",
            "2020-05-25 11:26:34,230 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:26:34,230 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:26:34,230 \tHypothesis: Ngubani uJosefa ebekazibuza ?\n",
            "2020-05-25 11:26:34,230 Validation result (greedy) at epoch  11, step     8000: bleu:  12.31, loss: 59846.4844, ppl:  32.5668, duration: 44.0985s\n",
            "2020-05-25 11:27:25,648 Epoch  11 Step:     8100 Batch Loss:     3.825439 Tokens per Sec:     3959, Lr: 0.000300\n",
            "2020-05-25 11:28:17,000 Epoch  11 Step:     8200 Batch Loss:     3.389385 Tokens per Sec:     3997, Lr: 0.000300\n",
            "2020-05-25 11:29:07,661 Epoch  11 Step:     8300 Batch Loss:     3.341794 Tokens per Sec:     3921, Lr: 0.000300\n",
            "2020-05-25 11:29:46,812 Epoch  11: total training loss 2508.36\n",
            "2020-05-25 11:29:46,813 EPOCH 12\n",
            "2020-05-25 11:29:58,701 Epoch  12 Step:     8400 Batch Loss:     3.539187 Tokens per Sec:     4036, Lr: 0.000300\n",
            "2020-05-25 11:30:49,794 Epoch  12 Step:     8500 Batch Loss:     3.257367 Tokens per Sec:     3895, Lr: 0.000300\n",
            "2020-05-25 11:31:40,601 Epoch  12 Step:     8600 Batch Loss:     2.892198 Tokens per Sec:     3957, Lr: 0.000300\n",
            "2020-05-25 11:32:30,986 Epoch  12 Step:     8700 Batch Loss:     3.448338 Tokens per Sec:     3904, Lr: 0.000300\n",
            "2020-05-25 11:33:22,662 Epoch  12 Step:     8800 Batch Loss:     3.046474 Tokens per Sec:     4004, Lr: 0.000300\n",
            "2020-05-25 11:34:13,419 Epoch  12 Step:     8900 Batch Loss:     2.132654 Tokens per Sec:     3937, Lr: 0.000300\n",
            "2020-05-25 11:35:04,631 Epoch  12 Step:     9000 Batch Loss:     2.975393 Tokens per Sec:     4019, Lr: 0.000300\n",
            "2020-05-25 11:35:45,621 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:35:45,621 Saving new checkpoint.\n",
            "2020-05-25 11:35:46,083 Example #0\n",
            "2020-05-25 11:35:46,084 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:35:46,084 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:35:46,084 \tHypothesis: Ikosi beyihlubuki , beyihlobanisa amakhosi wakwa - Israyeli bona enze izinto .\n",
            "2020-05-25 11:35:46,084 Example #1\n",
            "2020-05-25 11:35:46,084 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:35:46,084 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:35:46,084 \tHypothesis: U - Esewu nekakhetha bona kubayini umraro lo , watjela u - Isaka : “ Ningatshwenyeki , ngitjho nanyana mina mina mina mina mina mina mina mina nodadwethu . . . . . . . . . . . . . . . . . . . . . . .\n",
            "2020-05-25 11:35:46,084 Example #2\n",
            "2020-05-25 11:35:46,085 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:35:46,085 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:35:46,085 \tHypothesis: Nekafunda ukuthi indodakwakhe , bekafuna ukumtlhala , ngitjho nanyana ingilozi kaZimu yatjela uMariya bona enze lokho uMariya .\n",
            "2020-05-25 11:35:46,085 Example #3\n",
            "2020-05-25 11:35:46,085 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:35:46,085 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:35:46,086 \tHypothesis: Bobani uJosefa aba - Arimathiya ?\n",
            "2020-05-25 11:35:46,086 Validation result (greedy) at epoch  12, step     9000: bleu:  13.71, loss: 57646.3984, ppl:  28.6525, duration: 41.4540s\n",
            "2020-05-25 11:36:37,134 Epoch  12 Step:     9100 Batch Loss:     2.816390 Tokens per Sec:     3928, Lr: 0.000300\n",
            "2020-05-25 11:36:57,278 Epoch  12: total training loss 2423.78\n",
            "2020-05-25 11:36:57,278 EPOCH 13\n",
            "2020-05-25 11:37:27,245 Epoch  13 Step:     9200 Batch Loss:     2.518830 Tokens per Sec:     3927, Lr: 0.000300\n",
            "2020-05-25 11:38:18,233 Epoch  13 Step:     9300 Batch Loss:     3.324573 Tokens per Sec:     3885, Lr: 0.000300\n",
            "2020-05-25 11:39:10,053 Epoch  13 Step:     9400 Batch Loss:     2.894380 Tokens per Sec:     3933, Lr: 0.000300\n",
            "2020-05-25 11:40:01,800 Epoch  13 Step:     9500 Batch Loss:     3.528171 Tokens per Sec:     3984, Lr: 0.000300\n",
            "2020-05-25 11:40:52,664 Epoch  13 Step:     9600 Batch Loss:     3.334397 Tokens per Sec:     3916, Lr: 0.000300\n",
            "2020-05-25 11:41:44,049 Epoch  13 Step:     9700 Batch Loss:     2.807940 Tokens per Sec:     3939, Lr: 0.000300\n",
            "2020-05-25 11:42:34,762 Epoch  13 Step:     9800 Batch Loss:     3.769112 Tokens per Sec:     3997, Lr: 0.000300\n",
            "2020-05-25 11:43:25,777 Epoch  13 Step:     9900 Batch Loss:     3.474180 Tokens per Sec:     3975, Lr: 0.000300\n",
            "2020-05-25 11:43:26,862 Epoch  13: total training loss 2335.16\n",
            "2020-05-25 11:43:26,862 EPOCH 14\n",
            "2020-05-25 11:44:17,715 Epoch  14 Step:    10000 Batch Loss:     2.759861 Tokens per Sec:     3954, Lr: 0.000300\n",
            "2020-05-25 11:45:04,183 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:45:04,183 Saving new checkpoint.\n",
            "2020-05-25 11:45:04,634 Example #0\n",
            "2020-05-25 11:45:04,634 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:45:04,634 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:45:04,634 \tHypothesis: Ikosi yahlubukako beyirhagele khulu , yamakhosi wakwa - Israyeli ebekadosa phambili .\n",
            "2020-05-25 11:45:04,634 Example #1\n",
            "2020-05-25 11:45:04,635 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:45:04,635 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:45:04,635 \tHypothesis: U - Esewu nekavumela bona kubayini angikhetha isiqunto sokubeletha , wathi : “ Ningesabi , ngitjho nanyana mina mina mina mina mina mina mina mina mina mina mina mina mina mina , mina mina mina mina mina ! . . . . . . . . . . . . . . . .\n",
            "2020-05-25 11:45:04,635 Example #2\n",
            "2020-05-25 11:45:04,635 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:45:04,635 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:45:04,635 \tHypothesis: Nekafunda ngaye ekuthomeni , bekafuna ukuba nesineke , ngitjho nangaphambi kobana ingilozi kaZimu yamtjela bona uMariya .\n",
            "2020-05-25 11:45:04,636 Example #3\n",
            "2020-05-25 11:45:04,636 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:45:04,636 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:45:04,636 \tHypothesis: Ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:45:04,636 Validation result (greedy) at epoch  14, step    10000: bleu:  14.41, loss: 56374.8633, ppl:  26.6086, duration: 46.9210s\n",
            "2020-05-25 11:45:56,029 Epoch  14 Step:    10100 Batch Loss:     3.091419 Tokens per Sec:     3941, Lr: 0.000300\n",
            "2020-05-25 11:46:47,034 Epoch  14 Step:    10200 Batch Loss:     2.939920 Tokens per Sec:     3959, Lr: 0.000300\n",
            "2020-05-25 11:47:38,199 Epoch  14 Step:    10300 Batch Loss:     3.538213 Tokens per Sec:     4009, Lr: 0.000300\n",
            "2020-05-25 11:48:29,565 Epoch  14 Step:    10400 Batch Loss:     2.450259 Tokens per Sec:     3882, Lr: 0.000300\n",
            "2020-05-25 11:49:20,355 Epoch  14 Step:    10500 Batch Loss:     3.633758 Tokens per Sec:     3917, Lr: 0.000300\n",
            "2020-05-25 11:50:11,446 Epoch  14 Step:    10600 Batch Loss:     3.121220 Tokens per Sec:     3991, Lr: 0.000300\n",
            "2020-05-25 11:50:43,128 Epoch  14: total training loss 2265.29\n",
            "2020-05-25 11:50:43,128 EPOCH 15\n",
            "2020-05-25 11:51:01,843 Epoch  15 Step:    10700 Batch Loss:     3.182974 Tokens per Sec:     3857, Lr: 0.000300\n",
            "2020-05-25 11:51:52,874 Epoch  15 Step:    10800 Batch Loss:     2.115698 Tokens per Sec:     3966, Lr: 0.000300\n",
            "2020-05-25 11:52:44,473 Epoch  15 Step:    10900 Batch Loss:     3.158496 Tokens per Sec:     3984, Lr: 0.000300\n",
            "2020-05-25 11:53:35,603 Epoch  15 Step:    11000 Batch Loss:     2.826963 Tokens per Sec:     3959, Lr: 0.000300\n",
            "2020-05-25 11:54:12,735 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 11:54:12,735 Saving new checkpoint.\n",
            "2020-05-25 11:54:13,216 Example #0\n",
            "2020-05-25 11:54:13,216 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 11:54:13,216 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 11:54:13,217 \tHypothesis: Ikosi beyisolo ihlubuka , yagandelela amakhosi wakwa - Israyeli bona enze izinto ezimbi .\n",
            "2020-05-25 11:54:13,217 Example #1\n",
            "2020-05-25 11:54:13,217 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 11:54:13,217 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 11:54:13,217 \tHypothesis: U - Esewu nekabona bona khuyini eyikhetha nengabe angavumelani , wabawa u - Isaka wathi : “ Ningatshwenyeki , ngitjho nangami ! . . . . . . . . . .\n",
            "2020-05-25 11:54:13,217 Example #2\n",
            "2020-05-25 11:54:13,217 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 11:54:13,218 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 11:54:13,218 \tHypothesis: Nekafunda ngaye , bekafuna ukumsola , ngitjho nanyana ingilozi kaZimu yatjela uMariya bona enzeni uMariya .\n",
            "2020-05-25 11:54:13,218 Example #3\n",
            "2020-05-25 11:54:13,218 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 11:54:13,218 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 11:54:13,218 \tHypothesis: Ngubani uJosefa ebekakwenza u - Arimathiya ?\n",
            "2020-05-25 11:54:13,218 Validation result (greedy) at epoch  15, step    11000: bleu:  14.67, loss: 55355.2812, ppl:  25.0755, duration: 37.6155s\n",
            "2020-05-25 11:55:05,294 Epoch  15 Step:    11100 Batch Loss:     2.912613 Tokens per Sec:     3981, Lr: 0.000300\n",
            "2020-05-25 11:55:56,489 Epoch  15 Step:    11200 Batch Loss:     2.501546 Tokens per Sec:     3882, Lr: 0.000300\n",
            "2020-05-25 11:56:47,357 Epoch  15 Step:    11300 Batch Loss:     2.903274 Tokens per Sec:     3897, Lr: 0.000300\n",
            "2020-05-25 11:57:38,119 Epoch  15 Step:    11400 Batch Loss:     2.935089 Tokens per Sec:     3934, Lr: 0.000300\n",
            "2020-05-25 11:57:50,733 Epoch  15: total training loss 2200.65\n",
            "2020-05-25 11:57:50,733 EPOCH 16\n",
            "2020-05-25 11:58:29,710 Epoch  16 Step:    11500 Batch Loss:     2.318067 Tokens per Sec:     3918, Lr: 0.000300\n",
            "2020-05-25 11:59:21,015 Epoch  16 Step:    11600 Batch Loss:     2.116566 Tokens per Sec:     3961, Lr: 0.000300\n",
            "2020-05-25 12:00:12,457 Epoch  16 Step:    11700 Batch Loss:     1.657208 Tokens per Sec:     3922, Lr: 0.000300\n",
            "2020-05-25 12:01:03,611 Epoch  16 Step:    11800 Batch Loss:     2.232226 Tokens per Sec:     4020, Lr: 0.000300\n",
            "2020-05-25 12:01:54,720 Epoch  16 Step:    11900 Batch Loss:     2.733254 Tokens per Sec:     3850, Lr: 0.000300\n",
            "2020-05-25 12:02:46,319 Epoch  16 Step:    12000 Batch Loss:     2.933255 Tokens per Sec:     4009, Lr: 0.000300\n",
            "2020-05-25 12:03:33,510 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:03:33,510 Saving new checkpoint.\n",
            "2020-05-25 12:03:33,996 Example #0\n",
            "2020-05-25 12:03:33,996 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:03:33,996 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:03:33,997 \tHypothesis: Ikosi yahlubuka , inani elikhulu elikhulu lama - Israyeli lamakhosi ama - Israyeli wemvelo lebanga lokudlulisela .\n",
            "2020-05-25 12:03:33,997 Example #1\n",
            "2020-05-25 12:03:33,997 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:03:33,997 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:03:33,997 \tHypothesis: U - Esewu nekavumela bona into eyikhetha ngayo , wathi ku - Isaka : “ Tjhisi , ngitjho nanyana ubaba , nami nami angami . . . . .\n",
            "2020-05-25 12:03:33,997 Example #2\n",
            "2020-05-25 12:03:33,998 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:03:33,998 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:03:33,998 \tHypothesis: Nekafunda bona bekadlala , bekafuna ukuba nomusa kuye , ngitjho nangaphambi kobana ingilozi kaZimu yahlathulula lokho uMariya .\n",
            "2020-05-25 12:03:33,998 Example #3\n",
            "2020-05-25 12:03:33,998 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:03:33,998 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:03:33,998 \tHypothesis: Ngubani uJosefa ebekathiywa ngu - Arimathiya ?\n",
            "2020-05-25 12:03:33,998 Validation result (greedy) at epoch  16, step    12000: bleu:  15.46, loss: 54200.1680, ppl:  23.4450, duration: 47.6787s\n",
            "2020-05-25 12:04:25,498 Epoch  16 Step:    12100 Batch Loss:     3.026308 Tokens per Sec:     3927, Lr: 0.000300\n",
            "2020-05-25 12:05:07,786 Epoch  16: total training loss 2129.14\n",
            "2020-05-25 12:05:07,786 EPOCH 17\n",
            "2020-05-25 12:05:16,782 Epoch  17 Step:    12200 Batch Loss:     2.988366 Tokens per Sec:     3969, Lr: 0.000300\n",
            "2020-05-25 12:06:07,706 Epoch  17 Step:    12300 Batch Loss:     3.031974 Tokens per Sec:     4015, Lr: 0.000300\n",
            "2020-05-25 12:06:58,998 Epoch  17 Step:    12400 Batch Loss:     1.818121 Tokens per Sec:     3958, Lr: 0.000300\n",
            "2020-05-25 12:07:50,018 Epoch  17 Step:    12500 Batch Loss:     1.967185 Tokens per Sec:     3928, Lr: 0.000300\n",
            "2020-05-25 12:08:40,489 Epoch  17 Step:    12600 Batch Loss:     2.011937 Tokens per Sec:     3891, Lr: 0.000300\n",
            "2020-05-25 12:09:32,079 Epoch  17 Step:    12700 Batch Loss:     3.119825 Tokens per Sec:     3974, Lr: 0.000300\n",
            "2020-05-25 12:10:23,865 Epoch  17 Step:    12800 Batch Loss:     2.841220 Tokens per Sec:     4024, Lr: 0.000300\n",
            "2020-05-25 12:11:14,510 Epoch  17 Step:    12900 Batch Loss:     3.279654 Tokens per Sec:     3971, Lr: 0.000300\n",
            "2020-05-25 12:11:35,930 Epoch  17: total training loss 2081.47\n",
            "2020-05-25 12:11:35,930 EPOCH 18\n",
            "2020-05-25 12:12:05,489 Epoch  18 Step:    13000 Batch Loss:     2.676652 Tokens per Sec:     3923, Lr: 0.000300\n",
            "2020-05-25 12:12:54,432 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:12:54,432 Saving new checkpoint.\n",
            "2020-05-25 12:12:54,921 Example #0\n",
            "2020-05-25 12:12:54,922 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:12:54,922 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:12:54,922 \tHypothesis: Ikosi besele ihlongandlebe , isiqubuthu esikhulu samakhosi wakwa - Israyeli ebekanawo .\n",
            "2020-05-25 12:12:54,922 Example #1\n",
            "2020-05-25 12:12:54,922 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:12:54,923 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:12:54,923 \tHypothesis: U - Esewu nekavumela bona ngikuphi ukutjhigamela kwesizathu esizwakalako ebekambonileko , wabawa u - Isaka wathi : “ Akhahlamezi , ngitjho nomntwana ! .\n",
            "2020-05-25 12:12:54,923 Example #2\n",
            "2020-05-25 12:12:54,923 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:12:54,923 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:12:54,923 \tHypothesis: Nekafunda bona intombi le beyingamfazi , bekafuna bona aqalane nayo , ngitjho nangaphambi kobana ingilozi kaZimu yamtjela bona enzeni uMariya .\n",
            "2020-05-25 12:12:54,924 Example #3\n",
            "2020-05-25 12:12:54,924 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:12:54,924 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:12:54,924 \tHypothesis: Ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:12:54,924 Validation result (greedy) at epoch  18, step    13000: bleu:  16.09, loss: 53559.3516, ppl:  22.5867, duration: 49.4347s\n",
            "2020-05-25 12:13:45,661 Epoch  18 Step:    13100 Batch Loss:     3.206995 Tokens per Sec:     3941, Lr: 0.000300\n",
            "2020-05-25 12:14:37,119 Epoch  18 Step:    13200 Batch Loss:     2.392627 Tokens per Sec:     3947, Lr: 0.000300\n",
            "2020-05-25 12:15:27,641 Epoch  18 Step:    13300 Batch Loss:     2.687269 Tokens per Sec:     3931, Lr: 0.000300\n",
            "2020-05-25 12:16:18,740 Epoch  18 Step:    13400 Batch Loss:     2.552677 Tokens per Sec:     3941, Lr: 0.000300\n",
            "2020-05-25 12:17:09,947 Epoch  18 Step:    13500 Batch Loss:     2.722476 Tokens per Sec:     3935, Lr: 0.000300\n",
            "2020-05-25 12:18:01,461 Epoch  18 Step:    13600 Batch Loss:     2.929390 Tokens per Sec:     3970, Lr: 0.000300\n",
            "2020-05-25 12:18:52,275 Epoch  18 Step:    13700 Batch Loss:     3.004923 Tokens per Sec:     4000, Lr: 0.000300\n",
            "2020-05-25 12:18:54,351 Epoch  18: total training loss 2045.30\n",
            "2020-05-25 12:18:54,351 EPOCH 19\n",
            "2020-05-25 12:19:43,736 Epoch  19 Step:    13800 Batch Loss:     2.463118 Tokens per Sec:     3860, Lr: 0.000300\n",
            "2020-05-25 12:20:34,952 Epoch  19 Step:    13900 Batch Loss:     2.182680 Tokens per Sec:     3930, Lr: 0.000300\n",
            "2020-05-25 12:21:26,139 Epoch  19 Step:    14000 Batch Loss:     2.719429 Tokens per Sec:     3888, Lr: 0.000300\n",
            "2020-05-25 12:22:13,421 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:22:13,421 Saving new checkpoint.\n",
            "2020-05-25 12:22:13,886 Example #0\n",
            "2020-05-25 12:22:13,886 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:22:13,886 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:22:13,887 \tHypothesis: Ikosi besele ibekelwe ebugqilini , isiqubuthu esikhulu samakhosi wakwa - Israyeli bona enze izinto .\n",
            "2020-05-25 12:22:13,887 Example #1\n",
            "2020-05-25 12:22:13,887 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:22:13,887 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:22:13,887 \tHypothesis: U - Esewu nekalemuka bona khuyini engikhetha esikhundleni sokukhetha , wabawa u - Isaka wathi : “ Ngithabe nami , ngitjho nobaba . .\n",
            "2020-05-25 12:22:13,887 Example #2\n",
            "2020-05-25 12:22:13,888 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:22:13,888 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:22:13,888 \tHypothesis: Nekafunda bona bekanekankere , bekafuna ukuba nomusa kuye , ngitjho nanyana ingilozi kaZimu yatjela uMariya bona enze lokho okwatjhiwo nguMariya .\n",
            "2020-05-25 12:22:13,888 Example #3\n",
            "2020-05-25 12:22:13,888 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:22:13,888 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:22:13,888 \tHypothesis: Ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:22:13,889 Validation result (greedy) at epoch  19, step    14000: bleu:  16.30, loss: 52983.3711, ppl:  21.8420, duration: 47.7488s\n",
            "2020-05-25 12:23:04,625 Epoch  19 Step:    14100 Batch Loss:     2.533882 Tokens per Sec:     3929, Lr: 0.000300\n",
            "2020-05-25 12:23:55,608 Epoch  19 Step:    14200 Batch Loss:     2.797831 Tokens per Sec:     3929, Lr: 0.000300\n",
            "2020-05-25 12:24:47,154 Epoch  19 Step:    14300 Batch Loss:     2.804958 Tokens per Sec:     4029, Lr: 0.000300\n",
            "2020-05-25 12:25:38,598 Epoch  19 Step:    14400 Batch Loss:     2.673256 Tokens per Sec:     3956, Lr: 0.000300\n",
            "2020-05-25 12:26:12,575 Epoch  19: total training loss 1995.76\n",
            "2020-05-25 12:26:12,576 EPOCH 20\n",
            "2020-05-25 12:26:30,260 Epoch  20 Step:    14500 Batch Loss:     2.611196 Tokens per Sec:     3759, Lr: 0.000300\n",
            "2020-05-25 12:27:20,922 Epoch  20 Step:    14600 Batch Loss:     2.902875 Tokens per Sec:     3966, Lr: 0.000300\n",
            "2020-05-25 12:28:12,200 Epoch  20 Step:    14700 Batch Loss:     2.458294 Tokens per Sec:     3995, Lr: 0.000300\n",
            "2020-05-25 12:29:03,761 Epoch  20 Step:    14800 Batch Loss:     2.953134 Tokens per Sec:     3948, Lr: 0.000300\n",
            "2020-05-25 12:29:54,784 Epoch  20 Step:    14900 Batch Loss:     2.069063 Tokens per Sec:     3960, Lr: 0.000300\n",
            "2020-05-25 12:30:46,071 Epoch  20 Step:    15000 Batch Loss:     2.763923 Tokens per Sec:     3909, Lr: 0.000300\n",
            "2020-05-25 12:31:21,050 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:31:21,051 Saving new checkpoint.\n",
            "2020-05-25 12:31:21,549 Example #0\n",
            "2020-05-25 12:31:21,549 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:31:21,550 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:31:21,550 \tHypothesis: Ikosi besele ihlukanise iinhlubuki , ezikulu zamakhosi wakwa - Israyeli egade anawo iphuzu .\n",
            "2020-05-25 12:31:21,550 Example #1\n",
            "2020-05-25 12:31:21,550 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:31:21,550 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:31:21,550 \tHypothesis: U - Esewu nekabona bona khuyini ebeyingikhetha esikhundleni sokubeletha , wabawa u - Isaka wathi : “ Ngithwala , ngitjho nobaba ! . . . . . .\n",
            "2020-05-25 12:31:21,550 Example #2\n",
            "2020-05-25 12:31:21,551 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:31:21,551 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:31:21,551 \tHypothesis: Nekafunda bona bekasilingekile , bekafuna bona afune ukuba nomusa kuye , ngitjho nangaphambi kobana ingilozi kaZimu yamtjela bona enzeni uMariya .\n",
            "2020-05-25 12:31:21,551 Example #3\n",
            "2020-05-25 12:31:21,551 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:31:21,551 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:31:21,551 \tHypothesis: Bobani uJosefa aba - Arimathiya ?\n",
            "2020-05-25 12:31:21,551 Validation result (greedy) at epoch  20, step    15000: bleu:  16.71, loss: 52261.3750, ppl:  20.9432, duration: 35.4800s\n",
            "2020-05-25 12:32:12,952 Epoch  20 Step:    15100 Batch Loss:     3.054322 Tokens per Sec:     3986, Lr: 0.000300\n",
            "2020-05-25 12:33:03,808 Epoch  20 Step:    15200 Batch Loss:     2.870383 Tokens per Sec:     3906, Lr: 0.000300\n",
            "2020-05-25 12:33:17,789 Epoch  20: total training loss 1950.82\n",
            "2020-05-25 12:33:17,789 EPOCH 21\n",
            "2020-05-25 12:33:55,268 Epoch  21 Step:    15300 Batch Loss:     2.708786 Tokens per Sec:     3868, Lr: 0.000300\n",
            "2020-05-25 12:34:46,680 Epoch  21 Step:    15400 Batch Loss:     2.456804 Tokens per Sec:     4007, Lr: 0.000300\n",
            "2020-05-25 12:35:37,781 Epoch  21 Step:    15500 Batch Loss:     2.431135 Tokens per Sec:     3995, Lr: 0.000300\n",
            "2020-05-25 12:36:29,006 Epoch  21 Step:    15600 Batch Loss:     1.550877 Tokens per Sec:     3968, Lr: 0.000300\n",
            "2020-05-25 12:37:20,419 Epoch  21 Step:    15700 Batch Loss:     2.860752 Tokens per Sec:     3928, Lr: 0.000300\n",
            "2020-05-25 12:38:11,012 Epoch  21 Step:    15800 Batch Loss:     2.168457 Tokens per Sec:     3944, Lr: 0.000300\n",
            "2020-05-25 12:39:02,235 Epoch  21 Step:    15900 Batch Loss:     1.779843 Tokens per Sec:     3946, Lr: 0.000300\n",
            "2020-05-25 12:39:46,601 Epoch  21: total training loss 1908.61\n",
            "2020-05-25 12:39:46,602 EPOCH 22\n",
            "2020-05-25 12:39:53,533 Epoch  22 Step:    16000 Batch Loss:     2.541933 Tokens per Sec:     3867, Lr: 0.000300\n",
            "2020-05-25 12:40:32,270 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:40:32,270 Saving new checkpoint.\n",
            "2020-05-25 12:40:32,755 Example #0\n",
            "2020-05-25 12:40:32,755 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:40:32,755 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:40:32,755 \tHypothesis: Ikosi besele ibinzondo yobuqili , isiqubuthu esikhulu samakhosi wakwa - Israyeli bona ahlwe .\n",
            "2020-05-25 12:40:32,755 Example #1\n",
            "2020-05-25 12:40:32,756 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:40:32,756 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:40:32,756 \tHypothesis: U - Esewu nekabona bona ngisiphi isiqunto esingakahlakaniphi ebekambulali , wabawa u - Isaka wathi : “ Aspeli , ngitjho nanyana mina mina , mina nobaba . .\n",
            "2020-05-25 12:40:32,756 Example #2\n",
            "2020-05-25 12:40:32,756 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:40:32,757 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:40:32,757 \tHypothesis: Nekafunda bona bekanesibindi , bekafuna ukumhlathululele bona kubayini ingilozi kaZimu yamtjela bona kwenzekani kuJesu .\n",
            "2020-05-25 12:40:32,757 Example #3\n",
            "2020-05-25 12:40:32,757 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:40:32,757 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:40:32,757 \tHypothesis: Bobani uJosefa aba - Arimathiya ?\n",
            "2020-05-25 12:40:32,757 Validation result (greedy) at epoch  22, step    16000: bleu:  17.20, loss: 51425.2188, ppl:  19.9483, duration: 39.2242s\n",
            "2020-05-25 12:41:24,157 Epoch  22 Step:    16100 Batch Loss:     2.565468 Tokens per Sec:     3945, Lr: 0.000300\n",
            "2020-05-25 12:42:14,934 Epoch  22 Step:    16200 Batch Loss:     2.368915 Tokens per Sec:     3949, Lr: 0.000300\n",
            "2020-05-25 12:43:06,147 Epoch  22 Step:    16300 Batch Loss:     2.577751 Tokens per Sec:     3944, Lr: 0.000300\n",
            "2020-05-25 12:43:57,482 Epoch  22 Step:    16400 Batch Loss:     2.556015 Tokens per Sec:     4089, Lr: 0.000300\n",
            "2020-05-25 12:44:48,606 Epoch  22 Step:    16500 Batch Loss:     2.239263 Tokens per Sec:     3897, Lr: 0.000300\n",
            "2020-05-25 12:45:39,214 Epoch  22 Step:    16600 Batch Loss:     2.981003 Tokens per Sec:     3917, Lr: 0.000300\n",
            "2020-05-25 12:46:30,706 Epoch  22 Step:    16700 Batch Loss:     2.938588 Tokens per Sec:     4031, Lr: 0.000300\n",
            "2020-05-25 12:46:53,864 Epoch  22: total training loss 1866.05\n",
            "2020-05-25 12:46:53,864 EPOCH 23\n",
            "2020-05-25 12:47:21,488 Epoch  23 Step:    16800 Batch Loss:     2.617186 Tokens per Sec:     3870, Lr: 0.000300\n",
            "2020-05-25 12:48:12,070 Epoch  23 Step:    16900 Batch Loss:     2.579906 Tokens per Sec:     3906, Lr: 0.000300\n",
            "2020-05-25 12:49:03,638 Epoch  23 Step:    17000 Batch Loss:     2.368757 Tokens per Sec:     4014, Lr: 0.000300\n",
            "2020-05-25 12:49:33,537 Example #0\n",
            "2020-05-25 12:49:33,538 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:49:33,538 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:49:33,538 \tHypothesis: Ikosi besele ibinzinzinzondo , isiqubuthu esikhulu samakhosi wakwa - Israyeli bona ahlwe .\n",
            "2020-05-25 12:49:33,538 Example #1\n",
            "2020-05-25 12:49:33,538 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:49:33,538 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:49:33,539 \tHypothesis: U - Esewu nekabuzwa bona ngisiphi isiqunto esingakahlakaniphi , wabawa u - Isaka wathi : “ Ngithabe nami , ngitjho nobaba ! . . . .\n",
            "2020-05-25 12:49:33,539 Example #2\n",
            "2020-05-25 12:49:33,539 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:49:33,539 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:49:33,539 \tHypothesis: Wathi nakasesemntwana , bekafuna ukuba nomusa kuye , ngitjho nanyana ingilozi kaZimu yamhlathululela bona khuyini eyenzeka uMariya .\n",
            "2020-05-25 12:49:33,539 Example #3\n",
            "2020-05-25 12:49:33,540 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:49:33,540 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:49:33,540 \tHypothesis: Bobani uJosefa aba - Arimathiya ?\n",
            "2020-05-25 12:49:33,540 Validation result (greedy) at epoch  23, step    17000: bleu:  17.08, loss: 51554.6641, ppl:  20.0992, duration: 29.9015s\n",
            "2020-05-25 12:50:24,838 Epoch  23 Step:    17100 Batch Loss:     2.038454 Tokens per Sec:     3918, Lr: 0.000300\n",
            "2020-05-25 12:51:15,120 Epoch  23 Step:    17200 Batch Loss:     2.791324 Tokens per Sec:     4056, Lr: 0.000300\n",
            "2020-05-25 12:52:05,362 Epoch  23 Step:    17300 Batch Loss:     2.481578 Tokens per Sec:     3975, Lr: 0.000300\n",
            "2020-05-25 12:52:55,816 Epoch  23 Step:    17400 Batch Loss:     2.624002 Tokens per Sec:     4049, Lr: 0.000300\n",
            "2020-05-25 12:53:45,531 Epoch  23 Step:    17500 Batch Loss:     2.310919 Tokens per Sec:     4038, Lr: 0.000300\n",
            "2020-05-25 12:53:49,609 Epoch  23: total training loss 1842.98\n",
            "2020-05-25 12:53:49,609 EPOCH 24\n",
            "2020-05-25 12:54:36,353 Epoch  24 Step:    17600 Batch Loss:     2.417732 Tokens per Sec:     4023, Lr: 0.000300\n",
            "2020-05-25 12:55:26,259 Epoch  24 Step:    17700 Batch Loss:     2.605726 Tokens per Sec:     3975, Lr: 0.000300\n",
            "2020-05-25 12:56:16,194 Epoch  24 Step:    17800 Batch Loss:     2.440283 Tokens per Sec:     4031, Lr: 0.000300\n",
            "2020-05-25 12:57:06,803 Epoch  24 Step:    17900 Batch Loss:     2.609377 Tokens per Sec:     4050, Lr: 0.000300\n",
            "2020-05-25 12:57:57,345 Epoch  24 Step:    18000 Batch Loss:     2.638469 Tokens per Sec:     4021, Lr: 0.000300\n",
            "2020-05-25 12:58:29,935 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 12:58:29,935 Saving new checkpoint.\n",
            "2020-05-25 12:58:30,405 Example #0\n",
            "2020-05-25 12:58:30,405 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 12:58:30,405 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 12:58:30,405 \tHypothesis: Ikosi besele ihlinzinzinzinzinzinzwe , beyivela kwa - Israyeli bona enze njalo .\n",
            "2020-05-25 12:58:30,406 Example #1\n",
            "2020-05-25 12:58:30,406 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 12:58:30,406 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 12:58:30,406 \tHypothesis: U - Esewu nekabona bona ngisiphi isiqunto esingakahlakaniphi esamenza , wabawa u - Isaka : “ Ngithabe nami , ngitjho noyise , mina nobaba ! . . .\n",
            "2020-05-25 12:58:30,406 Example #2\n",
            "2020-05-25 12:58:30,406 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 12:58:30,407 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 12:58:30,407 \tHypothesis: Nekafunda bona bekanesibindi , bekafuna ukuba nomusa kuye , ngitjho nangaphambi kobana ingilozi kaZimu yamtjela bona kwenzekani uMariya .\n",
            "2020-05-25 12:58:30,407 Example #3\n",
            "2020-05-25 12:58:30,407 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 12:58:30,407 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 12:58:30,407 \tHypothesis: Bekuzokuba zokuba nguJosefa we - Arimathiya ?\n",
            "2020-05-25 12:58:30,407 Validation result (greedy) at epoch  24, step    18000: bleu:  17.72, loss: 50781.8320, ppl:  19.2151, duration: 33.0615s\n",
            "2020-05-25 12:59:20,160 Epoch  24 Step:    18100 Batch Loss:     2.178338 Tokens per Sec:     3977, Lr: 0.000300\n",
            "2020-05-25 13:00:10,758 Epoch  24 Step:    18200 Batch Loss:     2.687871 Tokens per Sec:     4013, Lr: 0.000300\n",
            "2020-05-25 13:00:46,002 Epoch  24: total training loss 1810.07\n",
            "2020-05-25 13:00:46,002 EPOCH 25\n",
            "2020-05-25 13:01:00,432 Epoch  25 Step:    18300 Batch Loss:     2.583940 Tokens per Sec:     3976, Lr: 0.000300\n",
            "2020-05-25 13:01:50,644 Epoch  25 Step:    18400 Batch Loss:     2.505003 Tokens per Sec:     4074, Lr: 0.000300\n",
            "2020-05-25 13:02:41,073 Epoch  25 Step:    18500 Batch Loss:     1.245688 Tokens per Sec:     3960, Lr: 0.000300\n",
            "2020-05-25 13:03:31,349 Epoch  25 Step:    18600 Batch Loss:     2.388033 Tokens per Sec:     4114, Lr: 0.000300\n",
            "2020-05-25 13:04:21,406 Epoch  25 Step:    18700 Batch Loss:     2.480107 Tokens per Sec:     3972, Lr: 0.000300\n",
            "2020-05-25 13:05:11,850 Epoch  25 Step:    18800 Batch Loss:     2.617943 Tokens per Sec:     4012, Lr: 0.000300\n",
            "2020-05-25 13:06:02,362 Epoch  25 Step:    18900 Batch Loss:     2.788568 Tokens per Sec:     3983, Lr: 0.000300\n",
            "2020-05-25 13:06:52,791 Epoch  25 Step:    19000 Batch Loss:     2.623625 Tokens per Sec:     3995, Lr: 0.000300\n",
            "2020-05-25 13:07:21,866 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 13:07:21,867 Saving new checkpoint.\n",
            "2020-05-25 13:07:22,347 Example #0\n",
            "2020-05-25 13:07:22,347 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 13:07:22,347 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 13:07:22,347 \tHypothesis: Ikosi besele ibinzinzinzwe , beyivela kwa - Israyeli bona ibe mamala .\n",
            "2020-05-25 13:07:22,347 Example #1\n",
            "2020-05-25 13:07:22,348 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 13:07:22,348 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 13:07:22,348 \tHypothesis: U - Esewu nekalemuka bona ngisiphi isiqunto esingakahlakaniphi ebekasenze , wabawa u - Isaka : “ Azange angilahle , ngitjho nobabami ! . .\n",
            "2020-05-25 13:07:22,348 Example #2\n",
            "2020-05-25 13:07:22,348 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 13:07:22,348 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 13:07:22,349 \tHypothesis: Nakafunda ukuthi bekambamba , bekafuna ukuba nomusa kuye , ngitjho nangaphambi kobana ingilozi kaZimu yamhlathululela bona kwenzekani uMariya .\n",
            "2020-05-25 13:07:22,349 Example #3\n",
            "2020-05-25 13:07:22,349 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 13:07:22,349 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 13:07:22,349 \tHypothesis: Bekuzokuba nguJosefa we - Arimathiya ?\n",
            "2020-05-25 13:07:22,349 Validation result (greedy) at epoch  25, step    19000: bleu:  17.59, loss: 50529.4297, ppl:  18.9349, duration: 29.5582s\n",
            "2020-05-25 13:07:38,635 Epoch  25: total training loss 1777.35\n",
            "2020-05-25 13:07:38,635 EPOCH 26\n",
            "2020-05-25 13:08:13,148 Epoch  26 Step:    19100 Batch Loss:     2.113364 Tokens per Sec:     3856, Lr: 0.000300\n",
            "2020-05-25 13:09:04,136 Epoch  26 Step:    19200 Batch Loss:     2.358621 Tokens per Sec:     4066, Lr: 0.000300\n",
            "2020-05-25 13:09:55,060 Epoch  26 Step:    19300 Batch Loss:     2.495302 Tokens per Sec:     4003, Lr: 0.000300\n",
            "2020-05-25 13:10:45,899 Epoch  26 Step:    19400 Batch Loss:     2.160303 Tokens per Sec:     3980, Lr: 0.000300\n",
            "2020-05-25 13:11:36,382 Epoch  26 Step:    19500 Batch Loss:     2.631365 Tokens per Sec:     3989, Lr: 0.000300\n",
            "2020-05-25 13:12:27,431 Epoch  26 Step:    19600 Batch Loss:     2.385973 Tokens per Sec:     4034, Lr: 0.000300\n",
            "2020-05-25 13:13:17,619 Epoch  26 Step:    19700 Batch Loss:     2.412109 Tokens per Sec:     4022, Lr: 0.000300\n",
            "2020-05-25 13:14:02,927 Epoch  26: total training loss 1741.40\n",
            "2020-05-25 13:14:02,928 EPOCH 27\n",
            "2020-05-25 13:14:08,204 Epoch  27 Step:    19800 Batch Loss:     2.666044 Tokens per Sec:     3872, Lr: 0.000300\n",
            "2020-05-25 13:14:58,137 Epoch  27 Step:    19900 Batch Loss:     2.714296 Tokens per Sec:     3952, Lr: 0.000300\n",
            "2020-05-25 13:15:47,851 Epoch  27 Step:    20000 Batch Loss:     2.449810 Tokens per Sec:     4024, Lr: 0.000300\n",
            "2020-05-25 13:16:48,815 Example #0\n",
            "2020-05-25 13:16:48,815 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 13:16:48,816 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 13:16:48,816 \tHypothesis: Ikosi besele ibinzinzinzondo , omumbi wakwa - Israyeli wadlulisela iphuzu elinjalo .\n",
            "2020-05-25 13:16:48,816 Example #1\n",
            "2020-05-25 13:16:48,816 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 13:16:48,816 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 13:16:48,816 \tHypothesis: U - Esewu nekalemuka bona ngisiphi isiqunto esingakahlakaniphi esenzileko , wabawa u - Isaka : “ Ngibawa ungivumele , ngitjho nobaba !\n",
            "2020-05-25 13:16:48,816 Example #2\n",
            "2020-05-25 13:16:48,817 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 13:16:48,817 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 13:16:48,817 \tHypothesis: Nekafunda bona bekambamba , bekafuna ukuba nomusa kuye , ngitjho nangaphambi kobana ingilozi kaZimu yahlathululela lokho egade kwenzeke uMariya .\n",
            "2020-05-25 13:16:48,817 Example #3\n",
            "2020-05-25 13:16:48,817 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 13:16:48,817 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 13:16:48,817 \tHypothesis: Bekuzokuba nguJosefa we - Arimathiya ?\n",
            "2020-05-25 13:16:48,817 Validation result (greedy) at epoch  27, step    20000: bleu:  17.91, loss: 50921.8477, ppl:  19.3723, duration: 60.9663s\n",
            "2020-05-25 13:17:39,426 Epoch  27 Step:    20100 Batch Loss:     2.674980 Tokens per Sec:     4142, Lr: 0.000300\n",
            "2020-05-25 13:18:29,578 Epoch  27 Step:    20200 Batch Loss:     1.698183 Tokens per Sec:     4071, Lr: 0.000300\n",
            "2020-05-25 13:19:19,265 Epoch  27 Step:    20300 Batch Loss:     2.509459 Tokens per Sec:     4028, Lr: 0.000300\n",
            "2020-05-25 13:20:09,686 Epoch  27 Step:    20400 Batch Loss:     2.503241 Tokens per Sec:     3967, Lr: 0.000300\n",
            "2020-05-25 13:21:00,024 Epoch  27 Step:    20500 Batch Loss:     2.531883 Tokens per Sec:     3988, Lr: 0.000300\n",
            "2020-05-25 13:21:26,254 Epoch  27: total training loss 1723.38\n",
            "2020-05-25 13:21:26,255 EPOCH 28\n",
            "2020-05-25 13:21:50,806 Epoch  28 Step:    20600 Batch Loss:     2.324357 Tokens per Sec:     3978, Lr: 0.000300\n",
            "2020-05-25 13:22:41,185 Epoch  28 Step:    20700 Batch Loss:     1.893052 Tokens per Sec:     4003, Lr: 0.000300\n",
            "2020-05-25 13:23:31,232 Epoch  28 Step:    20800 Batch Loss:     2.386988 Tokens per Sec:     4069, Lr: 0.000300\n",
            "2020-05-25 13:24:22,252 Epoch  28 Step:    20900 Batch Loss:     2.175425 Tokens per Sec:     3998, Lr: 0.000300\n",
            "2020-05-25 13:25:12,788 Epoch  28 Step:    21000 Batch Loss:     1.499287 Tokens per Sec:     3974, Lr: 0.000300\n",
            "2020-05-25 13:25:43,831 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 13:25:43,831 Saving new checkpoint.\n",
            "2020-05-25 13:25:44,260 Example #0\n",
            "2020-05-25 13:25:44,261 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 13:25:44,262 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 13:25:44,262 \tHypothesis: Ikosi besele ibogaboga , ubukhulu kwama - Israyeli egade banamandla wokwenza njalo .\n",
            "2020-05-25 13:25:44,262 Example #1\n",
            "2020-05-25 13:25:44,263 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 13:25:44,263 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 13:25:44,263 \tHypothesis: U - Esewu nekabona ukuthi ngisiphi isiqunto esithethe , wabawa u - Isaka wathi : “ Azange ngibalekele , ngitjho nobaba ! .\n",
            "2020-05-25 13:25:44,263 Example #2\n",
            "2020-05-25 13:25:44,263 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 13:25:44,263 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 13:25:44,264 \tHypothesis: Nekafunda bona bekambamba , bekafuna ukumsola , ngitjho nangaphambi kobana ingilozi kaZimu yahlathululela lokho okwenzeka kuMariya .\n",
            "2020-05-25 13:25:44,264 Example #3\n",
            "2020-05-25 13:25:44,264 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 13:25:44,264 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 13:25:44,264 \tHypothesis: Bekuzokuba nguJesefa we - Arimathiya ?\n",
            "2020-05-25 13:25:44,264 Validation result (greedy) at epoch  28, step    21000: bleu:  17.92, loss: 50338.4961, ppl:  18.7256, duration: 31.4759s\n",
            "2020-05-25 13:26:34,924 Epoch  28 Step:    21100 Batch Loss:     2.143458 Tokens per Sec:     3913, Lr: 0.000300\n",
            "2020-05-25 13:27:25,109 Epoch  28 Step:    21200 Batch Loss:     2.375255 Tokens per Sec:     3996, Lr: 0.000300\n",
            "2020-05-25 13:28:15,753 Epoch  28 Step:    21300 Batch Loss:     2.519253 Tokens per Sec:     4002, Lr: 0.000300\n",
            "2020-05-25 13:28:22,724 Epoch  28: total training loss 1699.08\n",
            "2020-05-25 13:28:22,724 EPOCH 29\n",
            "2020-05-25 13:29:06,084 Epoch  29 Step:    21400 Batch Loss:     2.282044 Tokens per Sec:     3989, Lr: 0.000300\n",
            "2020-05-25 13:29:56,932 Epoch  29 Step:    21500 Batch Loss:     2.258498 Tokens per Sec:     3968, Lr: 0.000300\n",
            "2020-05-25 13:30:46,707 Epoch  29 Step:    21600 Batch Loss:     1.852776 Tokens per Sec:     3974, Lr: 0.000300\n",
            "2020-05-25 13:31:36,574 Epoch  29 Step:    21700 Batch Loss:     2.184012 Tokens per Sec:     4015, Lr: 0.000300\n",
            "2020-05-25 13:32:26,802 Epoch  29 Step:    21800 Batch Loss:     2.440323 Tokens per Sec:     4053, Lr: 0.000300\n",
            "2020-05-25 13:33:16,663 Epoch  29 Step:    21900 Batch Loss:     2.422097 Tokens per Sec:     4021, Lr: 0.000300\n",
            "2020-05-25 13:34:06,754 Epoch  29 Step:    22000 Batch Loss:     1.386654 Tokens per Sec:     3994, Lr: 0.000300\n",
            "2020-05-25 13:34:41,836 Hooray! New best validation result [ppl]!\n",
            "2020-05-25 13:34:41,836 Saving new checkpoint.\n",
            "2020-05-25 13:34:42,298 Example #0\n",
            "2020-05-25 13:34:42,298 \tSource:     The king was already an apostate , the worst of Israel’s kings up to that point .\n",
            "2020-05-25 13:34:42,298 \tReference:  Ikosi besele isihlubuki , iyimbi ukuwadlula woke amakhosi wakwa - Israyeli ngesikhatheso .\n",
            "2020-05-25 13:34:42,298 \tHypothesis: Ikosi besele ibinzondo yobubuki , amakhosi wakwa - Israyeli wona wona wona wona bekalinga ngamandla ukwenza njalo .\n",
            "2020-05-25 13:34:42,298 Example #1\n",
            "2020-05-25 13:34:42,299 \tSource:     When Esau belatedly realized what a foolish choice he had made , he begged Isaac : “ Bless me , even me too , my father ! . . .\n",
            "2020-05-25 13:34:42,299 \tReference:  U - Esewu nekalemuka ngemva kwesikhathi bona wenze isiqunto sobudlhayela , wancenga u - Isaka wathi : “ Nami ngibusisa baba ! . . .\n",
            "2020-05-25 13:34:42,299 \tHypothesis: U - Esewu nekabona bona khuyini okweqa esikhundleni sakhe sokukhetha , wabawa u - Isaka wathi : “ Ngibawa ungilibalele ngitjho nanyana nobaba ! .\n",
            "2020-05-25 13:34:42,299 Example #2\n",
            "2020-05-25 13:34:42,299 \tSource:     When he first learned that she was pregnant , he wanted to deal mercifully with her , even before God’s angel explained to him what had happened to Mary .\n",
            "2020-05-25 13:34:42,299 \tReference:  Indaba yokobana uMariya usidisi neyifika eendlebeni zakhe , wafuna ukumphatha ngomusa , ngitjho nangaphambi kobana ingilozi kaZimu imhlathululele okwenzeke kuMariya .\n",
            "2020-05-25 13:34:42,300 \tHypothesis: Nekafunda bona bekambamba , bekafuna ukumkhalima , ngitjho nangaphambi kobana ingilozi kaZimu yamtjela bona kwenzekeni uMariya .\n",
            "2020-05-25 13:34:42,300 Example #3\n",
            "2020-05-25 13:34:42,300 \tSource:     Who was this Joseph of Arimathea ?\n",
            "2020-05-25 13:34:42,300 \tReference:  Kanti ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 13:34:42,300 \tHypothesis: Bekuzokuba ngubani uJosefa we - Arimathiya ?\n",
            "2020-05-25 13:34:42,300 Validation result (greedy) at epoch  29, step    22000: bleu:  18.91, loss: 49742.1797, ppl:  18.0869, duration: 35.5456s\n",
            "2020-05-25 13:35:22,573 Epoch  29: total training loss 1685.50\n",
            "2020-05-25 13:35:22,573 EPOCH 30\n",
            "2020-05-25 13:35:32,535 Epoch  30 Step:    22100 Batch Loss:     1.949780 Tokens per Sec:     4038, Lr: 0.000300\n",
            "2020-05-25 13:36:23,245 Epoch  30 Step:    22200 Batch Loss:     2.507748 Tokens per Sec:     4015, Lr: 0.000300\n",
            "2020-05-25 13:37:12,665 Epoch  30 Step:    22300 Batch Loss:     2.382967 Tokens per Sec:     3978, Lr: 0.000300\n",
            "2020-05-25 13:38:03,790 Epoch  30 Step:    22400 Batch Loss:     2.338923 Tokens per Sec:     3987, Lr: 0.000300\n",
            "2020-05-25 13:38:54,832 Epoch  30 Step:    22500 Batch Loss:     2.305876 Tokens per Sec:     4000, Lr: 0.000300\n",
            "2020-05-25 13:39:44,953 Epoch  30 Step:    22600 Batch Loss:     1.948185 Tokens per Sec:     3901, Lr: 0.000300\n",
            "2020-05-25 13:40:35,337 Epoch  30 Step:    22700 Batch Loss:     2.519622 Tokens per Sec:     4002, Lr: 0.000300\n",
            "2020-05-25 13:41:26,393 Epoch  30 Step:    22800 Batch Loss:     1.205953 Tokens per Sec:     4001, Lr: 0.000300\n",
            "2020-05-25 13:41:47,730 Epoch  30: total training loss 1648.48\n",
            "2020-05-25 13:41:47,730 Training ended after  30 epochs.\n",
            "2020-05-25 13:41:47,730 Best validation result (greedy) at step    22000:  18.09 ppl.\n",
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n",
            "2020-05-25 13:42:37,959  dev bleu:  19.77 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-05-25 13:42:37,960 Translations saved to: models/ennr_transformer/00022000.hyps.dev\n",
            "2020-05-25 13:43:47,198 test bleu:  26.61 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-05-25 13:43:47,201 Translations saved to: models/ennr_transformer/00022000.hyps.test\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "outputId": "046b29d7-db5a-49e0-9098-09e9154c8d31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Copy the created models from the notebook storage to google drive for persistant storage \n",
        "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\""
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot create symbolic link '/content/drive/My Drive/masakhane/en-nr-baseline/models/ennr_transformer/best.ckpt': Operation not supported\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4WeRIo2_66Jq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp /content/joeynmt/models/ennr_transformer/best.ckpt \"/content/drive/My Drive/masakhane/en-nr-baseline/models/ennr_transformer/\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "outputId": "1387224e-c7ff-4ba8-d80e-5d4dcfca6872",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 386
        }
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 1000\tLoss: 99971.88281\tPPL: 336.54996\tbleu: 0.12180\tLR: 0.00030000\t*\n",
            "Steps: 2000\tLoss: 87661.05469\tPPL: 164.38464\tbleu: 1.24824\tLR: 0.00030000\t*\n",
            "Steps: 3000\tLoss: 78894.19531\tPPL: 98.68610\tbleu: 3.11364\tLR: 0.00030000\t*\n",
            "Steps: 4000\tLoss: 73069.34375\tPPL: 70.31013\tbleu: 5.35443\tLR: 0.00030000\t*\n",
            "Steps: 5000\tLoss: 68052.57031\tPPL: 52.50563\tbleu: 8.08551\tLR: 0.00030000\t*\n",
            "Steps: 6000\tLoss: 64672.61719\tPPL: 43.12893\tbleu: 10.06278\tLR: 0.00030000\t*\n",
            "Steps: 7000\tLoss: 61825.87109\tPPL: 36.54347\tbleu: 11.18306\tLR: 0.00030000\t*\n",
            "Steps: 8000\tLoss: 59846.48438\tPPL: 32.56684\tbleu: 12.30515\tLR: 0.00030000\t*\n",
            "Steps: 9000\tLoss: 57646.39844\tPPL: 28.65252\tbleu: 13.71154\tLR: 0.00030000\t*\n",
            "Steps: 10000\tLoss: 56374.86328\tPPL: 26.60856\tbleu: 14.40518\tLR: 0.00030000\t*\n",
            "Steps: 11000\tLoss: 55355.28125\tPPL: 25.07545\tbleu: 14.66825\tLR: 0.00030000\t*\n",
            "Steps: 12000\tLoss: 54200.16797\tPPL: 23.44500\tbleu: 15.46316\tLR: 0.00030000\t*\n",
            "Steps: 13000\tLoss: 53559.35156\tPPL: 22.58666\tbleu: 16.08570\tLR: 0.00030000\t*\n",
            "Steps: 14000\tLoss: 52983.37109\tPPL: 21.84201\tbleu: 16.29692\tLR: 0.00030000\t*\n",
            "Steps: 15000\tLoss: 52261.37500\tPPL: 20.94316\tbleu: 16.71145\tLR: 0.00030000\t*\n",
            "Steps: 16000\tLoss: 51425.21875\tPPL: 19.94832\tbleu: 17.19772\tLR: 0.00030000\t*\n",
            "Steps: 17000\tLoss: 51554.66406\tPPL: 20.09918\tbleu: 17.08415\tLR: 0.00030000\t\n",
            "Steps: 18000\tLoss: 50781.83203\tPPL: 19.21511\tbleu: 17.71864\tLR: 0.00030000\t*\n",
            "Steps: 19000\tLoss: 50529.42969\tPPL: 18.93489\tbleu: 17.59382\tLR: 0.00030000\t*\n",
            "Steps: 20000\tLoss: 50921.84766\tPPL: 19.37235\tbleu: 17.90990\tLR: 0.00030000\t\n",
            "Steps: 21000\tLoss: 50338.49609\tPPL: 18.72563\tbleu: 17.92365\tLR: 0.00030000\t*\n",
            "Steps: 22000\tLoss: 49742.17969\tPPL: 18.08685\tbleu: 18.90914\tLR: 0.00030000\t*\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "outputId": "37117975-25db-4c0e-9a38-cf8251d904f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-05-25 13:46:02,382 Hello! This is Joey-NMT.\n",
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n",
            "2020-05-25 13:46:54,063  dev bleu:  19.77 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-05-25 13:48:02,692 test bleu:  26.61 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "foeaIXVo75Pe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}