{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "jw300_ve_starter_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "958cac4a-aba4-41f1-cd7b-b45ee64b9f9f"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"ve\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2d5de188-921c-423f-f61b-6c7c4c201dc2"
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-ve-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "51927cb3-2837-425c-edb3-77ddfc38b527"
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 15.0MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 4.3MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 5.5MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 5.8MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 4.9MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 5.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 3.9MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222
        },
        "outputId": "e96c579f-5e6b-442b-f9c4-8e8bc7510cb4"
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-ve.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            "   2 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-ve.xml.gz\n",
            " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
            "  25 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/ve.zip\n",
            "\n",
            " 290 MB Total size\n",
            "./JW300_latest_xml_en-ve.xml.gz ... 100% of 2 MB\n",
            "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
            "./JW300_latest_xml_ve.zip ... 100% of 25 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n48GDRnP8y2G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 591
        },
        "outputId": "78d16deb-0a07-4409-f67b-5ecae9bc9fd1"
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-05-19 08:55:26--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-05-19 08:55:26 (11.2 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2020-05-19 08:55:29--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ve.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 206464 (202K) [text/plain]\n",
            "Saving to: ‘test.en-ve.en’\n",
            "\n",
            "test.en-ve.en       100%[===================>] 201.62K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-05-19 08:55:30 (9.02 MB/s) - ‘test.en-ve.en’ saved [206464/206464]\n",
            "\n",
            "--2020-05-19 08:55:36--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ve.ve\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 245271 (240K) [text/plain]\n",
            "Saving to: ‘test.en-ve.ve’\n",
            "\n",
            "test.en-ve.ve       100%[===================>] 239.52K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-05-19 08:55:37 (10.0 MB/s) - ‘test.en-ve.ve’ saved [245271/245271]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NqDG-CI28y2L",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "756c1832-0c48-40a5-a1d0-035afb90f175"
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        },
        "outputId": "66e0dc15-5c0a-470f-a1b8-09f8224fb68e"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 5573/229715 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>‘ God , Why Did You Allow This ? ’</td>\n",
              "      <td>‘ Mudzimu , Ndi Ngani Wo Tendela Zwenezwi Zwit...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>RICARDO still remembers sitting with his wife ...</td>\n",
              "      <td>RICARDO u kha ḓi humbula o dzula na musadzi w...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>* Neither of them had the courage to read the ...</td>\n",
              "      <td>* Vhoṱhe vho vha vho tshuwa musi vho lindela u...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0                 ‘ God , Why Did You Allow This ? ’  ‘ Mudzimu , Ndi Ngani Wo Tendela Zwenezwi Zwit...\n",
              "1  RICARDO still remembers sitting with his wife ...  RICARDO u kha ḓi humbula o dzula na musadzi w...\n",
              "2  * Neither of them had the courage to read the ...  * Vhoṱhe vho vha vho tshuwa musi vho lindela u..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "outputId": "d3e457ed-20f7-43bd-93af-7cb6757e4078"
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Z_1BwAApEtMk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "a9335da0-848b-4616-c78b-9952307d5975"
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "from os import cpu_count\n",
        "from functools import partial\n",
        "from multiprocessing import Pool\n",
        "\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: fuzzywuzzy in /usr/local/lib/python3.6/dist-packages (0.18.0)\n",
            "Requirement already satisfied: python-Levenshtein in /usr/local/lib/python3.6/dist-packages (0.12.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (46.3.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5cCxB4ewIpMH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "outputId": "69b01629-61ef-4e27-e883-9c1c8ea36294"
      },
      "source": [
        "start_time = time.time()\n",
        "### iterating over pandas dataframe rows is not recomended, let use multi processing to apply the function\n",
        "\n",
        "with Pool(cpu_count()-1) as pool:\n",
        "    scores = pool.map(partial(fuzzfilter, candidates=list(en_test_sents), pad=5), df_pp['source_sentence'])\n",
        "hours, rem = divmod(time.time() - start_time, 3600)\n",
        "minutes, seconds = divmod(rem, 60)\n",
        "print(\"done in {}h:{}min:{}seconds\".format(hours, minutes, seconds))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp = df_pp.assign(scores=scores)\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇩']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '”']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "done in 1.0h:22.0min:1.7638554573059082seconds\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 810
        },
        "outputId": "0837b787-6314-4ba2-cf1a-8923096d5893"
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "Before long , a mob arrived , bearing torches and armed with swords and clubs .\n",
            "If so , our words and deeds will show that we are walking and living by spirit . ​ — Read Galatians 5 : 16 , 25 .\n",
            "With Jehovah’s backing , Joshua was able to lead God’s people into the Promised Land and to victory over all their enemies .\n",
            "Isaiah explains the spiritual consequences of immoderate drinking : “ The activity of Jehovah they do not look at , and the work of his hands they have not seen . ” ​ — Isaiah 5 : 11 , 12 .\n",
            "As we examine those words , let us keep in mind that they also apply to God’s worshippers today . ​ — Romans 15 : 4 .\n",
            "( a ) What challenges do many single Christians face ?\n",
            "He built houses and had gardens , orchards , servants , livestock , male and female singers , along with much gold and silver .\n",
            "If he had tolerated it , the well - being of all living creatures ​ — in heaven and on earth — ​ would be threatened .\n",
            "“ But there was no voice , and there was no one answering , ” the Bible says .\n",
            "So the prophet unequivocally replies : “ If you gave me half of your house I would not come with you and eat bread or drink water in this place . ”\n",
            "\n",
            "==> train.ve <==\n",
            "Nga u ṱavhanya ho mbo ḓi swika gogo ḽa vhathu , ḽo fara dzimbone nahone ḽo ḓiṱama nga mabanga na mafumo .\n",
            "Arali ri tshi u rabelela , zwine ra zwi amba na zwine ra zwi ita zwi ḓo sumbedza uri ri khou tshimbila na u tshila nga nḓila ya lwa muya . — Vhalani Vha - Galata 5 : 16 , 25 .\n",
            "31 : 6 ) Nga thuso ya Yehova , Yoshua o kona u ranga phanḓa vhathu vha Mudzimu u swika Shangoni ḽo Fulufhedziswaho na u kunda maswina avho oṱhe .\n",
            "Yesaya o ṱalutshedza mvelelo dzi si dzavhuḓi dza u nwesa musi a tshi ri : “ Zwine Yehova a zwi ita a vha zwi sedzi ; mishumo ya zwanḓa zwawe a vha i vhoni . ” — Yesaya 5 : 11 , 12 .\n",
            "Musi ri tshi ṱolisisa eneo maipfi , ri fanela u dzula ri tshi humbula uri a a shuma na kha vhagwadameli vha Mudzimu vha ṋamusi . — Vha - Roma 15 : 4 .\n",
            "( a ) Vhakriste vhanzhi vha siho mbinganoni vha sedzana na vhuleme vhufhio ?\n",
            "O fhaṱa dzinnḓu , a vha na dzingade , tsimu dza mitshelo , vhalanda , zwifuwo , vhaimbi vha vhanna na vha vhasadzi , u katela na musuku na tsimbi - tseṱha nnzhi vhukuma .\n",
            "( Hab . 1 : 13 ) Arali o vha o konḓelela zwenezwo , zwivhumbwa zwawe zwoṱhe zwi re ṱaḓulu na kha ḽifhasi , zwo vha zwi nga si thetshelese .\n",
            "Bivhili ri : “ Hu si vhe na phindulo na we a aravha . ”\n",
            "Ngauralo , muporofita o fhindula o khwaṱha a ri : “ Naho wa nkovhela zwa muṱani wau nga vhukati , a thi nga yi na iwe .\n",
            "==> dev.en <==\n",
            "Jehovah empowered Jesus to perform miracles .\n",
            "He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "“ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "14 , 15 . ( a ) What example did Jesus set in choosing the right companions ?\n",
            "18 What Does It Mean to Be a Spiritual Person ?\n",
            "One day a woman named Mrs . Ukoli said to one of Bintu’s friends : “ Bring the woman who lost two children at the same time and is still preaching the Bible’s message .\n",
            "How surprised she was to learn that I was the person who had initially given the book to her uncle when I met him in Kaunas ! ​ — Eccl .\n",
            "When giving advice , what principles should we keep in mind ?\n",
            "How can this help us as public instructors of the good news ?\n",
            "\n",
            "==> dev.ve <==\n",
            "Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "“ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "14 , 15 . ( a ) Yesu o ri vhetshela tsumbo ifhio ya u khetha khonani dzavhuḓi ?\n",
            "18 Zwi Amba Mini U Vha Muthu Ane A Livhiswa Nga Muya Wa Mudzimu ?\n",
            "Ḽiṅwe ḓuvha vhaṅwe mufumakadzi vhane vha pfi Vho - Ukoli vho vhudza muṅwe wa khonani dza Bintu vha ri : “ Mmbidzeleni houḽa musadzi o lovhelwaho nga vhana vhavhili ane na zwino a kha ḓi bvela phanḓa na u huwelela mulaedza wa Bivhili .\n",
            "O vha o mangala vhukuma musi a tshi pfa uri ho vha hu nṋe we nda ṋea khotsi muhulu wawe bugu yeneyo musi ndi tshi ṱangana navho ngei Kaunas ! — Muh .\n",
            "Ndi maitele afhio ane ra fanela u a humbula musi ri tshi ṋea nyeletshedzo ?\n",
            "Zwenezwi zwi nga ri thusa hani sa vhahuweleli vha mafhungo maḓifha ?\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "fb764e81-73a6-48f8-a4fb-6558a94dcbe0"
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 14, done.\u001b[K\n",
            "remote: Counting objects:   7% (1/14)\u001b[K\rremote: Counting objects:  14% (2/14)\u001b[K\rremote: Counting objects:  21% (3/14)\u001b[K\rremote: Counting objects:  28% (4/14)\u001b[K\rremote: Counting objects:  35% (5/14)\u001b[K\rremote: Counting objects:  42% (6/14)\u001b[K\rremote: Counting objects:  50% (7/14)\u001b[K\rremote: Counting objects:  57% (8/14)\u001b[K\rremote: Counting objects:  64% (9/14)\u001b[K\rremote: Counting objects:  71% (10/14)\u001b[K\rremote: Counting objects:  78% (11/14)\u001b[K\rremote: Counting objects:  85% (12/14)\u001b[K\rremote: Counting objects:  92% (13/14)\u001b[K\rremote: Counting objects: 100% (14/14)\u001b[K\rremote: Counting objects: 100% (14/14), done.\u001b[K\n",
            "remote: Compressing objects: 100% (13/13), done.\u001b[K\n",
            "remote: Total 2409 (delta 3), reused 4 (delta 1), pack-reused 2395\u001b[K\n",
            "Receiving objects: 100% (2409/2409), 2.63 MiB | 4.94 MiB/s, done.\n",
            "Resolving deltas: 100% (1682/1682), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (7.0.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.18.4)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (46.3.0)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.5.0+cu101)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6e/9d/9846507837ca50ae20917f59d83b79246b8313bd19d4f5bf575ecb98132b/sacrebleu-1.4.9-py3-none-any.whl (60kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 3.5MB/s \n",
            "\u001b[?25hCollecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.10.1)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 17.5MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/37/6e/36419ec1bd2208e157dff7fc3e565b185394c0dc4901e9e2f983cb1d4b7f/pylint-2.5.2-py3-none-any.whl (324kB)\n",
            "\u001b[K     |████████████████████████████████| 327kB 40.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Collecting wrapt==1.11.1\n",
            "  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
            "Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.1)\n",
            "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.0)\n",
            "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.10.0)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.3.3)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.34.2)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.6.3)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.28.1)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.4.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.41.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.23.0)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/53/84/7b3146ec6378d28abc73ab484f09f47dfa008ad6f03f33d90a369f880e25/portalocker-1.7.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.8.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.2.0)\n",
            "Requirement already satisfied: pandas>=0.22.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.0.3)\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 7.2MB/s \n",
            "\u001b[?25hCollecting toml>=0.7.1\n",
            "  Downloading https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl\n",
            "Collecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Collecting astroid<=2.5,>=2.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/46/c9/e9c2642dfb169590fb8bdb395f9329da042ee559c2ae7c1e612a3e5f40b4/astroid-2.4.1-py3-none-any.whl (214kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 41.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.7.2)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.0.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.6.0.post3)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.1)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.9)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2020.4.5.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.22.0->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 8.1MB/s \n",
            "\u001b[?25hCollecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 42.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (4.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.3.0)\n",
            "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Building wheels for collected packages: joeynmt, pyyaml, wrapt\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=73768 sha256=8b60e2f40d897162b06fb00603aa7836d411bbace2e02908d131a05b89ebb71c\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-z_bz47v3/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44621 sha256=4a41392aeeb4b752cdde4d36ac006814010e58e5cf3d6609973daabf05a96294\n",
            "  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
            "  Building wheel for wrapt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wrapt: filename=wrapt-1.11.1-cp36-cp36m-linux_x86_64.whl size=67430 sha256=d9fa5cfc57446ae3fbd049e7b4871482bd3ff47b5c9b4aa183b75b74beef1af8\n",
            "  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc\n",
            "Successfully built joeynmt pyyaml wrapt\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, isort, toml, mccabe, wrapt, lazy-object-proxy, typed-ast, astroid, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Found existing installation: wrapt 1.12.1\n",
            "    Uninstalling wrapt-1.12.1:\n",
            "      Successfully uninstalled wrapt-1.12.1\n",
            "Successfully installed astroid-2.4.1 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.7.0 pylint-2.5.2 pyyaml-5.3.1 sacrebleu-1.4.9 subword-nmt-0.3.7 toml-0.10.1 typed-ast-1.4.1 wrapt-1.11.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "outputId": "4aeaff23-b684-4fea-9ed7-ea7bd7faa7ce"
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 40000 -o bpe.codes.40000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.40000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.40000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.40000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Tshivenda Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.40000  dev.en       test.bpe.ve     test.ve\t    train.en\n",
            "dev.bpe.en\t dev.ve       test.en\t      train.bpe.en  train.ve\n",
            "dev.bpe.ve\t test.bpe.en  test.en-any.en  train.bpe.ve\n",
            "bpe.codes.40000  dev.en       test.bpe.ve     test.ve\t    train.en\n",
            "dev.bpe.en\t dev.ve       test.en\t      train.bpe.en  train.ve\n",
            "dev.bpe.ve\t test.bpe.en  test.en-any.en  train.bpe.ve\n",
            "BPE Tshivenda Sentences\n",
            "Nga zwenezwo , ndo vha ndi tshi ḓivhelwa u sa fulufhedzea .\n",
            "Musi ndi tshi guda ngoho , ndo laṱa danda na ṋowa naho mushumo wo vha u tshi holela vhukuma .\n",
            "Ndi tsumbo yavhuḓi kha vharwa vhanga vhavhili nahone ndi a fanelea u vha na ndugelo tshivhidzoni .\n",
            "Zwino vha@@ ṱ@@ oli vha mithelo na vhaṅwe vhane nda ita navho zwa mabindu vha nḓivha sa muthu a fulufhedzeaho . ”\n",
            "Ruthe o pfulutshela Isiraele he a kona u gwadamela Mudzimu wa ngoho .\n",
            "Combined BPE Vocab\n",
            "autom@@\n",
            "complet@@\n",
            "Mist@@\n",
            "reated\n",
            "Samarit@@\n",
            "ʺ\n",
            "returne@@\n",
            "omes\n",
            "itorium\n",
            "byterian\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlMitUHR8Qy-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "83c24b61-e287-4fca-f04c-f0621527de12"
      },
      "source": [
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.40000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\""
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.40000  dev.en       test.bpe.ve     test.ve\t    train.en\n",
            "dev.bpe.en\t dev.ve       test.en\t      train.bpe.en  train.ve\n",
            "dev.bpe.ve\t test.bpe.en  test.en-any.en  train.bpe.ve\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iprRFmA1e_cn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#THIS WAS NEVER SAVED TO DRIVE.\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py \"/content/drive/My Drive/masakhane/en-ve-baseline/train.bpe.$src\" \"/content/drive/My Drive/masakhane/en-ve-baseline/train.bpe.$tgt\" --output_path \"/content/drive/My Drive/masakhane/en-ve-baseline/vocab.txt\"\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"data/{name}/train.bpe\"\n",
        "    dev:   \"data/{name}/dev.bpe\"\n",
        "    test:  \"data/{name}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"data/{name}/vocab.txt\"\n",
        "    trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 30                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "5094c76a-125e-4360-bb33-0bccd8db1074"
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-05-19 10:32:08,756 Hello! This is Joey-NMT.\n",
            "2020-05-19 10:32:08.886563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-05-19 10:32:10,745 Total params: 20619776\n",
            "2020-05-19 10:32:10,747 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2020-05-19 10:32:24,862 cfg.name                           : enve_transformer\n",
            "2020-05-19 10:32:24,862 cfg.data.src                       : en\n",
            "2020-05-19 10:32:24,862 cfg.data.trg                       : ve\n",
            "2020-05-19 10:32:24,863 cfg.data.train                     : data/enve/train.bpe\n",
            "2020-05-19 10:32:24,863 cfg.data.dev                       : data/enve/dev.bpe\n",
            "2020-05-19 10:32:24,863 cfg.data.test                      : data/enve/test.bpe\n",
            "2020-05-19 10:32:24,863 cfg.data.level                     : bpe\n",
            "2020-05-19 10:32:24,863 cfg.data.lowercase                 : False\n",
            "2020-05-19 10:32:24,863 cfg.data.max_sent_length           : 100\n",
            "2020-05-19 10:32:24,863 cfg.data.src_vocab                 : data/enve/vocab.txt\n",
            "2020-05-19 10:32:24,863 cfg.data.trg_vocab                 : data/enve/vocab.txt\n",
            "2020-05-19 10:32:24,863 cfg.testing.beam_size              : 5\n",
            "2020-05-19 10:32:24,864 cfg.testing.alpha                  : 1.0\n",
            "2020-05-19 10:32:24,864 cfg.training.random_seed           : 42\n",
            "2020-05-19 10:32:24,864 cfg.training.optimizer             : adam\n",
            "2020-05-19 10:32:24,864 cfg.training.normalization         : tokens\n",
            "2020-05-19 10:32:24,864 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2020-05-19 10:32:24,864 cfg.training.scheduling            : plateau\n",
            "2020-05-19 10:32:24,864 cfg.training.patience              : 5\n",
            "2020-05-19 10:32:24,864 cfg.training.learning_rate_factor  : 0.5\n",
            "2020-05-19 10:32:24,864 cfg.training.learning_rate_warmup  : 1000\n",
            "2020-05-19 10:32:24,865 cfg.training.decrease_factor       : 0.7\n",
            "2020-05-19 10:32:24,865 cfg.training.loss                  : crossentropy\n",
            "2020-05-19 10:32:24,865 cfg.training.learning_rate         : 0.0003\n",
            "2020-05-19 10:32:24,865 cfg.training.learning_rate_min     : 1e-08\n",
            "2020-05-19 10:32:24,865 cfg.training.weight_decay          : 0.0\n",
            "2020-05-19 10:32:24,865 cfg.training.label_smoothing       : 0.1\n",
            "2020-05-19 10:32:24,865 cfg.training.batch_size            : 4096\n",
            "2020-05-19 10:32:24,865 cfg.training.batch_type            : token\n",
            "2020-05-19 10:32:24,865 cfg.training.eval_batch_size       : 3600\n",
            "2020-05-19 10:32:24,866 cfg.training.eval_batch_type       : token\n",
            "2020-05-19 10:32:24,866 cfg.training.batch_multiplier      : 1\n",
            "2020-05-19 10:32:24,866 cfg.training.early_stopping_metric : ppl\n",
            "2020-05-19 10:32:24,866 cfg.training.epochs                : 30\n",
            "2020-05-19 10:32:24,866 cfg.training.validation_freq       : 1000\n",
            "2020-05-19 10:32:24,866 cfg.training.logging_freq          : 100\n",
            "2020-05-19 10:32:24,866 cfg.training.eval_metric           : bleu\n",
            "2020-05-19 10:32:24,866 cfg.training.model_dir             : models/enve_transformer\n",
            "2020-05-19 10:32:24,866 cfg.training.overwrite             : False\n",
            "2020-05-19 10:32:24,866 cfg.training.shuffle               : True\n",
            "2020-05-19 10:32:24,867 cfg.training.use_cuda              : True\n",
            "2020-05-19 10:32:24,867 cfg.training.max_output_length     : 100\n",
            "2020-05-19 10:32:24,867 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2020-05-19 10:32:24,867 cfg.training.keep_last_ckpts       : 3\n",
            "2020-05-19 10:32:24,867 cfg.model.initializer              : xavier\n",
            "2020-05-19 10:32:24,867 cfg.model.bias_initializer         : zeros\n",
            "2020-05-19 10:32:24,867 cfg.model.init_gain                : 1.0\n",
            "2020-05-19 10:32:24,867 cfg.model.embed_initializer        : xavier\n",
            "2020-05-19 10:32:24,867 cfg.model.embed_init_gain          : 1.0\n",
            "2020-05-19 10:32:24,867 cfg.model.tied_embeddings          : True\n",
            "2020-05-19 10:32:24,868 cfg.model.tied_softmax             : True\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.type             : transformer\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.num_layers       : 6\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.num_heads        : 4\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.embeddings.scale : True\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.hidden_size      : 256\n",
            "2020-05-19 10:32:24,868 cfg.model.encoder.ff_size          : 1024\n",
            "2020-05-19 10:32:24,869 cfg.model.encoder.dropout          : 0.3\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.type             : transformer\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.num_layers       : 6\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.num_heads        : 4\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.embeddings.scale : True\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.hidden_size      : 256\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.ff_size          : 1024\n",
            "2020-05-19 10:32:24,869 cfg.model.decoder.dropout          : 0.3\n",
            "2020-05-19 10:32:24,870 Data set sizes: \n",
            "\ttrain 204153,\n",
            "\tvalid 1000,\n",
            "\ttest 2719\n",
            "2020-05-19 10:32:24,870 First training example:\n",
            "\t[SRC] Before long , a mob arrived , bearing torches and armed with swords and clubs .\n",
            "\t[TRG] Nga u ṱavhanya ho mbo ḓi swika gogo ḽa vhathu , ḽo fara dzi@@ mbone nahone ḽo ḓiṱama nga mabanga na mafumo .\n",
            "2020-05-19 10:32:24,870 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) a (7) u (8) vha (9) the\n",
            "2020-05-19 10:32:24,870 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) a (7) u (8) vha (9) the\n",
            "2020-05-19 10:32:24,871 Number of Src words (types): 37342\n",
            "2020-05-19 10:32:24,871 Number of Trg words (types): 37342\n",
            "2020-05-19 10:32:24,871 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=37342),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=37342))\n",
            "2020-05-19 10:32:24,902 EPOCH 1\n",
            "/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:\n",
            "\tnonzero(Tensor input, *, Tensor out)\n",
            "Consider using one of the following signatures instead:\n",
            "\tnonzero(Tensor input, *, bool as_tuple)\n",
            "2020-05-19 10:33:13,626 Epoch   1 Step:      100 Batch Loss:     5.510985 Tokens per Sec:     4577, Lr: 0.000300\n",
            "2020-05-19 10:34:01,802 Epoch   1 Step:      200 Batch Loss:     5.208167 Tokens per Sec:     4654, Lr: 0.000300\n",
            "2020-05-19 10:34:48,967 Epoch   1 Step:      300 Batch Loss:     5.170606 Tokens per Sec:     4520, Lr: 0.000300\n",
            "2020-05-19 10:35:37,902 Epoch   1 Step:      400 Batch Loss:     4.744290 Tokens per Sec:     4654, Lr: 0.000300\n",
            "2020-05-19 10:36:26,420 Epoch   1 Step:      500 Batch Loss:     5.057566 Tokens per Sec:     4735, Lr: 0.000300\n",
            "2020-05-19 10:37:15,400 Epoch   1 Step:      600 Batch Loss:     4.725267 Tokens per Sec:     4621, Lr: 0.000300\n",
            "2020-05-19 10:38:03,564 Epoch   1 Step:      700 Batch Loss:     4.522420 Tokens per Sec:     4494, Lr: 0.000300\n",
            "2020-05-19 10:38:51,753 Epoch   1 Step:      800 Batch Loss:     4.274016 Tokens per Sec:     4606, Lr: 0.000300\n",
            "2020-05-19 10:39:39,934 Epoch   1 Step:      900 Batch Loss:     4.417595 Tokens per Sec:     4504, Lr: 0.000300\n",
            "2020-05-19 10:40:27,942 Epoch   1 Step:     1000 Batch Loss:     4.745903 Tokens per Sec:     4674, Lr: 0.000300\n",
            "2020-05-19 10:42:44,994 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 10:42:44,994 Saving new checkpoint.\n",
            "2020-05-19 10:42:45,522 Example #0\n",
            "2020-05-19 10:42:45,522 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 10:42:45,522 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 10:42:45,522 \tHypothesis: Nga zwenezwo , u vha tshi vha tshi vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou ita .\n",
            "2020-05-19 10:42:45,522 Example #1\n",
            "2020-05-19 10:42:45,523 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 10:42:45,523 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 10:42:45,523 \tHypothesis: Nga zwenezwo , u vha tshi vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou ita u vha tshi khou ita u vha tshi ita u vha tshi ita u vha tshi khou ita u ita u ita u vha tshi khou ita\n",
            "2020-05-19 10:42:45,523 Example #2\n",
            "2020-05-19 10:42:45,523 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 10:42:45,523 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 10:42:45,523 \tHypothesis: Nga zwenezwo , u vha tshi vha tshi vha tshi vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi khou vha tshi vha tshi khou vha tshi khou khou khou khou khou khou khou khou khou khou khou\n",
            "2020-05-19 10:42:45,524 Example #3\n",
            "2020-05-19 10:42:45,524 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 10:42:45,524 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 10:42:45,524 \tHypothesis: U vha : “ Ndi mini nga u vha tshi khou ita uri ri tshi khou ita uri ri tshi khou ita uri ri tshi khou ita uri ri tshi khou ita u vha tshi khou ita u vha tshi khou ita nga u ita u vha tshi ita nga u vha tshi ita nga u vha tshi khou ita nga u vha tshi ita nga u vha tshi ita u vha tshi ita nga u vha tshi khou ita nga u vha tshi ita u vha tshi khou ita u ita u ita u ita u ita u ita\n",
            "2020-05-19 10:42:45,524 Validation result (greedy) at epoch   1, step     1000: bleu:   0.18, loss: 97914.3672, ppl:  66.6256, duration: 137.5819s\n",
            "2020-05-19 10:43:33,886 Epoch   1 Step:     1100 Batch Loss:     4.156778 Tokens per Sec:     4599, Lr: 0.000300\n",
            "2020-05-19 10:44:22,424 Epoch   1 Step:     1200 Batch Loss:     3.933909 Tokens per Sec:     4528, Lr: 0.000300\n",
            "2020-05-19 10:45:10,952 Epoch   1 Step:     1300 Batch Loss:     4.174785 Tokens per Sec:     4686, Lr: 0.000300\n",
            "2020-05-19 10:45:59,702 Epoch   1 Step:     1400 Batch Loss:     4.141608 Tokens per Sec:     4712, Lr: 0.000300\n",
            "2020-05-19 10:46:47,803 Epoch   1 Step:     1500 Batch Loss:     4.146492 Tokens per Sec:     4612, Lr: 0.000300\n",
            "2020-05-19 10:47:36,427 Epoch   1 Step:     1600 Batch Loss:     3.599254 Tokens per Sec:     4698, Lr: 0.000300\n",
            "2020-05-19 10:48:24,596 Epoch   1 Step:     1700 Batch Loss:     4.106578 Tokens per Sec:     4670, Lr: 0.000300\n",
            "2020-05-19 10:49:13,538 Epoch   1 Step:     1800 Batch Loss:     3.711218 Tokens per Sec:     4737, Lr: 0.000300\n",
            "2020-05-19 10:50:02,148 Epoch   1 Step:     1900 Batch Loss:     3.773057 Tokens per Sec:     4640, Lr: 0.000300\n",
            "2020-05-19 10:50:50,746 Epoch   1 Step:     2000 Batch Loss:     3.660550 Tokens per Sec:     4574, Lr: 0.000300\n",
            "2020-05-19 10:53:09,218 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 10:53:09,219 Saving new checkpoint.\n",
            "2020-05-19 10:53:09,679 Example #0\n",
            "2020-05-19 10:53:09,680 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 10:53:09,680 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 10:53:09,680 \tHypothesis: Yehova o vha a tshi ḓo vha a tshi khou vha na tshivhindi .\n",
            "2020-05-19 10:53:09,680 Example #1\n",
            "2020-05-19 10:53:09,680 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 10:53:09,680 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 10:53:09,680 \tHypothesis: O vha a tshi khou fanela u vha na vhungoho ha uri a tshi ḓo vha na u vha na vhungoho ha uri a tshi khou ita uri a vhe na u ita . — Vha - Heberu 1 : 1 .\n",
            "2020-05-19 10:53:09,681 Example #2\n",
            "2020-05-19 10:53:09,681 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 10:53:09,681 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 10:53:09,681 \tHypothesis: Nga murahu ha miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya miṅwaha ya 20 , o vha a tshi khou khou vha na tshivhindi .\n",
            "2020-05-19 10:53:09,682 Example #3\n",
            "2020-05-19 10:53:09,682 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 10:53:09,682 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 10:53:09,682 \tHypothesis: “ Ndi mini zwine zwa ḓo zwi ita uri ni kone u vha na u vha na dzangalelo kha ḽifhasi . ” — 1 : 1 - 15 .\n",
            "2020-05-19 10:53:09,682 Validation result (greedy) at epoch   1, step     2000: bleu:   2.45, loss: 77647.3750, ppl:  27.9365, duration: 138.9362s\n",
            "2020-05-19 10:53:57,974 Epoch   1 Step:     2100 Batch Loss:     3.568706 Tokens per Sec:     4650, Lr: 0.000300\n",
            "2020-05-19 10:53:59,958 Epoch   1: total training loss 9127.51\n",
            "2020-05-19 10:53:59,958 EPOCH 2\n",
            "2020-05-19 10:54:46,894 Epoch   2 Step:     2200 Batch Loss:     3.423502 Tokens per Sec:     4640, Lr: 0.000300\n",
            "2020-05-19 10:55:35,248 Epoch   2 Step:     2300 Batch Loss:     3.237967 Tokens per Sec:     4577, Lr: 0.000300\n",
            "2020-05-19 10:56:24,104 Epoch   2 Step:     2400 Batch Loss:     3.310958 Tokens per Sec:     4599, Lr: 0.000300\n",
            "2020-05-19 10:57:13,045 Epoch   2 Step:     2500 Batch Loss:     3.091684 Tokens per Sec:     4677, Lr: 0.000300\n",
            "2020-05-19 10:58:01,940 Epoch   2 Step:     2600 Batch Loss:     2.932214 Tokens per Sec:     4557, Lr: 0.000300\n",
            "2020-05-19 10:58:50,032 Epoch   2 Step:     2700 Batch Loss:     3.419168 Tokens per Sec:     4539, Lr: 0.000300\n",
            "2020-05-19 10:59:38,673 Epoch   2 Step:     2800 Batch Loss:     3.385809 Tokens per Sec:     4727, Lr: 0.000300\n",
            "2020-05-19 11:00:27,195 Epoch   2 Step:     2900 Batch Loss:     3.177925 Tokens per Sec:     4535, Lr: 0.000300\n",
            "2020-05-19 11:01:15,412 Epoch   2 Step:     3000 Batch Loss:     2.936191 Tokens per Sec:     4598, Lr: 0.000300\n",
            "2020-05-19 11:03:17,644 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:03:17,644 Saving new checkpoint.\n",
            "2020-05-19 11:03:18,114 Example #0\n",
            "2020-05-19 11:03:18,114 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:03:18,114 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:03:18,114 \tHypothesis: Yehova o vha a tshi khou ṱoḓa u ita uri a vhe na fulufhelo ḽa u vha na fulufhelo .\n",
            "2020-05-19 11:03:18,114 Example #1\n",
            "2020-05-19 11:03:18,115 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:03:18,115 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:03:18,115 \tHypothesis: O vha a tshi khou vha na vhuleme , nahone a vha a tshi khou ṱoḓa u vha na vhushaka havhuḓi na u vha na u vha na lutendo . — 1 Pet .\n",
            "2020-05-19 11:03:18,115 Example #2\n",
            "2020-05-19 11:03:18,115 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:03:18,115 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:03:18,115 \tHypothesis: Musi a tshi thoma u thoma , tshigwada tsha ngei United States , o vha a tshi khou fanela u vha na dzangalelo ḽa u vha na tshivhindi .\n",
            "2020-05-19 11:03:18,115 Example #3\n",
            "2020-05-19 11:03:18,116 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:03:18,116 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:03:18,116 \tHypothesis: “ Ni ḓo vha na mvelelo dzifhio dzine dza ḓo dzi wana dzi tshi bva kha ḓi - ḓe ? ” — 1 : 1 .\n",
            "2020-05-19 11:03:18,116 Validation result (greedy) at epoch   2, step     3000: bleu:   6.81, loss: 66852.9609, ppl:  17.5844, duration: 122.7034s\n",
            "2020-05-19 11:04:06,633 Epoch   2 Step:     3100 Batch Loss:     2.664735 Tokens per Sec:     4677, Lr: 0.000300\n",
            "2020-05-19 11:04:55,240 Epoch   2 Step:     3200 Batch Loss:     2.568233 Tokens per Sec:     4571, Lr: 0.000300\n",
            "2020-05-19 11:05:43,694 Epoch   2 Step:     3300 Batch Loss:     2.763402 Tokens per Sec:     4653, Lr: 0.000300\n",
            "2020-05-19 11:06:31,892 Epoch   2 Step:     3400 Batch Loss:     2.592956 Tokens per Sec:     4560, Lr: 0.000300\n",
            "2020-05-19 11:07:20,463 Epoch   2 Step:     3500 Batch Loss:     2.823352 Tokens per Sec:     4631, Lr: 0.000300\n",
            "2020-05-19 11:08:09,294 Epoch   2 Step:     3600 Batch Loss:     2.754943 Tokens per Sec:     4638, Lr: 0.000300\n",
            "2020-05-19 11:08:57,412 Epoch   2 Step:     3700 Batch Loss:     2.616460 Tokens per Sec:     4589, Lr: 0.000300\n",
            "2020-05-19 11:09:45,442 Epoch   2 Step:     3800 Batch Loss:     2.656302 Tokens per Sec:     4567, Lr: 0.000300\n",
            "2020-05-19 11:10:33,425 Epoch   2 Step:     3900 Batch Loss:     2.339253 Tokens per Sec:     4675, Lr: 0.000300\n",
            "2020-05-19 11:11:22,009 Epoch   2 Step:     4000 Batch Loss:     3.381377 Tokens per Sec:     4665, Lr: 0.000300\n",
            "2020-05-19 11:12:56,822 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:12:56,822 Saving new checkpoint.\n",
            "2020-05-19 11:12:57,313 Example #0\n",
            "2020-05-19 11:12:57,314 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:12:57,314 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:12:57,314 \tHypothesis: Yehova o ita uri Yesu o vha a tshi khou ṱoḓa u ita vhuṱolo .\n",
            "2020-05-19 11:12:57,314 Example #1\n",
            "2020-05-19 11:12:57,314 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:12:57,314 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:12:57,314 \tHypothesis: O vha a tshi khou ita phetho , fhedzi o vha a sa athu u vha a tshi khou ṱoḓa u vha na mushumo wawe . — 2 Sam .\n",
            "2020-05-19 11:12:57,315 Example #2\n",
            "2020-05-19 11:12:57,315 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:12:57,315 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:12:57,315 \tHypothesis: Nga murahu ha miṅwaha ya rathi , tshivhidzo tsha ḓana ḽa miṅwaha ḽa 15 ḽa 15 ḽa 15 ḽa 15 , o mbo ḓi vha a tshi khou ṱoḓa u vha na dzangalelo kha mushumo wa u vha na vhuḓifhinduleli ha u vha na vhuḓifhinduleli ha u vha na vhuḓifhinduleli .\n",
            "2020-05-19 11:12:57,315 Example #3\n",
            "2020-05-19 11:12:57,315 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:12:57,315 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:12:57,316 \tHypothesis: “ Ni ḓo vha hani u vha na mulandu wa u vha na mvelelo dzifhio ? ” — 1 Vha - Roma 14 : 14 .\n",
            "2020-05-19 11:12:57,316 Validation result (greedy) at epoch   2, step     4000: bleu:  10.57, loss: 58730.9648, ppl:  12.4124, duration: 95.3062s\n",
            "2020-05-19 11:13:45,878 Epoch   2 Step:     4100 Batch Loss:     2.484316 Tokens per Sec:     4722, Lr: 0.000300\n",
            "2020-05-19 11:14:34,035 Epoch   2 Step:     4200 Batch Loss:     3.220546 Tokens per Sec:     4633, Lr: 0.000300\n",
            "2020-05-19 11:14:37,468 Epoch   2: total training loss 6294.77\n",
            "2020-05-19 11:14:37,468 EPOCH 3\n",
            "2020-05-19 11:15:22,842 Epoch   3 Step:     4300 Batch Loss:     2.836008 Tokens per Sec:     4653, Lr: 0.000300\n",
            "2020-05-19 11:16:11,535 Epoch   3 Step:     4400 Batch Loss:     3.075299 Tokens per Sec:     4591, Lr: 0.000300\n",
            "2020-05-19 11:17:00,018 Epoch   3 Step:     4500 Batch Loss:     2.595468 Tokens per Sec:     4545, Lr: 0.000300\n",
            "2020-05-19 11:17:48,048 Epoch   3 Step:     4600 Batch Loss:     2.262842 Tokens per Sec:     4648, Lr: 0.000300\n",
            "2020-05-19 11:18:36,690 Epoch   3 Step:     4700 Batch Loss:     2.719151 Tokens per Sec:     4679, Lr: 0.000300\n",
            "2020-05-19 11:19:25,432 Epoch   3 Step:     4800 Batch Loss:     2.233371 Tokens per Sec:     4690, Lr: 0.000300\n",
            "2020-05-19 11:20:14,077 Epoch   3 Step:     4900 Batch Loss:     1.999152 Tokens per Sec:     4567, Lr: 0.000300\n",
            "2020-05-19 11:21:02,877 Epoch   3 Step:     5000 Batch Loss:     2.238546 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 11:22:22,749 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:22:22,749 Saving new checkpoint.\n",
            "2020-05-19 11:22:23,232 Example #0\n",
            "2020-05-19 11:22:23,233 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:22:23,233 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:22:23,233 \tHypothesis: Yehova o ṋea vhuṱanzi ha uri Yesu u a ita vhuṱolo .\n",
            "2020-05-19 11:22:23,233 Example #1\n",
            "2020-05-19 11:22:23,233 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:22:23,233 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:22:23,234 \tHypothesis: O konḓelela vhuleme ho raloho , fhedzi o vha a sa athu u vha a sa athu u fhisea kana u vha na muya wawe . — 2 Vha - Kor .\n",
            "2020-05-19 11:22:23,234 Example #2\n",
            "2020-05-19 11:22:23,234 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:22:23,234 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:22:23,234 \tHypothesis: Kha ḓana ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Roma tsho rumela Timotheo uri a kone u ṱhogomela ṱhoḓea ya Paulo .\n",
            "2020-05-19 11:22:23,234 Example #3\n",
            "2020-05-19 11:22:23,235 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:22:23,235 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:22:23,235 \tHypothesis: “ Ni ḓo vha hani na mulandu wa u wana mafhungo - lunzhi - lunzhi - lunzhi - lunzhi - lunzhi - lunzhi - ngoho . ” — 1 : 18 .\n",
            "2020-05-19 11:22:23,235 Validation result (greedy) at epoch   3, step     5000: bleu:  16.37, loss: 52724.7734, ppl:   9.5938, duration: 80.3570s\n",
            "2020-05-19 11:23:10,823 Epoch   3 Step:     5100 Batch Loss:     2.692603 Tokens per Sec:     4600, Lr: 0.000300\n",
            "2020-05-19 11:23:59,226 Epoch   3 Step:     5200 Batch Loss:     2.956306 Tokens per Sec:     4644, Lr: 0.000300\n",
            "2020-05-19 11:24:47,534 Epoch   3 Step:     5300 Batch Loss:     2.483693 Tokens per Sec:     4563, Lr: 0.000300\n",
            "2020-05-19 11:25:35,638 Epoch   3 Step:     5400 Batch Loss:     2.214419 Tokens per Sec:     4563, Lr: 0.000300\n",
            "2020-05-19 11:26:24,555 Epoch   3 Step:     5500 Batch Loss:     2.481165 Tokens per Sec:     4688, Lr: 0.000300\n",
            "2020-05-19 11:27:13,042 Epoch   3 Step:     5600 Batch Loss:     2.057866 Tokens per Sec:     4590, Lr: 0.000300\n",
            "2020-05-19 11:28:00,990 Epoch   3 Step:     5700 Batch Loss:     2.644410 Tokens per Sec:     4549, Lr: 0.000300\n",
            "2020-05-19 11:28:49,095 Epoch   3 Step:     5800 Batch Loss:     2.163410 Tokens per Sec:     4725, Lr: 0.000300\n",
            "2020-05-19 11:29:37,608 Epoch   3 Step:     5900 Batch Loss:     2.109244 Tokens per Sec:     4540, Lr: 0.000300\n",
            "2020-05-19 11:30:26,036 Epoch   3 Step:     6000 Batch Loss:     2.834045 Tokens per Sec:     4710, Lr: 0.000300\n",
            "2020-05-19 11:31:46,652 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:31:46,652 Saving new checkpoint.\n",
            "2020-05-19 11:31:47,129 Example #0\n",
            "2020-05-19 11:31:47,129 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:31:47,130 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:31:47,130 \tHypothesis: Yehova o ṋea vhuṱolo ha vhuṱolo .\n",
            "2020-05-19 11:31:47,130 Example #1\n",
            "2020-05-19 11:31:47,130 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:31:47,130 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:31:47,130 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi o vha a sa athu u fhisea kana u ita zwithu zwawe . — 2 Vhakor .\n",
            "2020-05-19 11:31:47,130 Example #2\n",
            "2020-05-19 11:31:47,131 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:31:47,131 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:31:47,131 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Timotheo uri a kone u ṱhogomela ṱhoḓea ya Paulo .\n",
            "2020-05-19 11:31:47,131 Example #3\n",
            "2020-05-19 11:31:47,131 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:31:47,131 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:31:47,131 \tHypothesis: “ Ni ḓo vha ni tshi ḓo vha na mulandu wo fhamba - fhambanaho nga nḓila - fhambanaho ? ” — 1 VHA - Roma 18 : 18 .\n",
            "2020-05-19 11:31:47,131 Validation result (greedy) at epoch   3, step     6000: bleu:  19.93, loss: 48624.9531, ppl:   8.0470, duration: 81.0945s\n",
            "2020-05-19 11:32:35,384 Epoch   3 Step:     6100 Batch Loss:     2.426149 Tokens per Sec:     4572, Lr: 0.000300\n",
            "2020-05-19 11:33:24,019 Epoch   3 Step:     6200 Batch Loss:     2.061434 Tokens per Sec:     4615, Lr: 0.000300\n",
            "2020-05-19 11:34:12,470 Epoch   3 Step:     6300 Batch Loss:     1.926799 Tokens per Sec:     4557, Lr: 0.000300\n",
            "2020-05-19 11:34:19,237 Epoch   3: total training loss 5125.89\n",
            "2020-05-19 11:34:19,237 EPOCH 4\n",
            "2020-05-19 11:35:01,272 Epoch   4 Step:     6400 Batch Loss:     2.631964 Tokens per Sec:     4552, Lr: 0.000300\n",
            "2020-05-19 11:35:49,804 Epoch   4 Step:     6500 Batch Loss:     2.021032 Tokens per Sec:     4645, Lr: 0.000300\n",
            "2020-05-19 11:36:37,668 Epoch   4 Step:     6600 Batch Loss:     1.984634 Tokens per Sec:     4452, Lr: 0.000300\n",
            "2020-05-19 11:37:26,084 Epoch   4 Step:     6700 Batch Loss:     1.861274 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 11:38:14,449 Epoch   4 Step:     6800 Batch Loss:     2.302662 Tokens per Sec:     4618, Lr: 0.000300\n",
            "2020-05-19 11:39:02,693 Epoch   4 Step:     6900 Batch Loss:     2.153662 Tokens per Sec:     4541, Lr: 0.000300\n",
            "2020-05-19 11:39:51,210 Epoch   4 Step:     7000 Batch Loss:     1.810007 Tokens per Sec:     4669, Lr: 0.000300\n",
            "2020-05-19 11:40:47,888 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:40:47,889 Saving new checkpoint.\n",
            "2020-05-19 11:40:48,353 Example #0\n",
            "2020-05-19 11:40:48,354 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:40:48,354 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:40:48,354 \tHypothesis: Yehova o ṋea Yesu vhuṱolo .\n",
            "2020-05-19 11:40:48,354 Example #1\n",
            "2020-05-19 11:40:48,354 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:40:48,354 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:40:48,354 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi o vha a sa athu u fhisetshela kana u sa fhelela hawe . — 2 Vhakor .\n",
            "2020-05-19 11:40:48,355 Example #2\n",
            "2020-05-19 11:40:48,355 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:40:48,355 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:40:48,355 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Timotheo uri a ṱoḓe u ṱhogomela ṱhoḓea dza Paulo .\n",
            "2020-05-19 11:40:48,355 Example #3\n",
            "2020-05-19 11:40:48,355 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:40:48,355 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:40:48,356 \tHypothesis: “ Ni ḓo ḽa hani u ḽa na u ḽa na u ḽa na mihumbulo mivhili mivhili mivhili ” — 1 / 18 .\n",
            "2020-05-19 11:40:48,356 Validation result (greedy) at epoch   4, step     7000: bleu:  23.20, loss: 45252.9531, ppl:   6.9635, duration: 57.1451s\n",
            "2020-05-19 11:41:37,096 Epoch   4 Step:     7100 Batch Loss:     1.873284 Tokens per Sec:     4660, Lr: 0.000300\n",
            "2020-05-19 11:42:26,244 Epoch   4 Step:     7200 Batch Loss:     2.143654 Tokens per Sec:     4667, Lr: 0.000300\n",
            "2020-05-19 11:43:14,852 Epoch   4 Step:     7300 Batch Loss:     2.074540 Tokens per Sec:     4534, Lr: 0.000300\n",
            "2020-05-19 11:44:03,109 Epoch   4 Step:     7400 Batch Loss:     1.967843 Tokens per Sec:     4619, Lr: 0.000300\n",
            "2020-05-19 11:44:51,659 Epoch   4 Step:     7500 Batch Loss:     2.631477 Tokens per Sec:     4671, Lr: 0.000300\n",
            "2020-05-19 11:45:39,488 Epoch   4 Step:     7600 Batch Loss:     1.915626 Tokens per Sec:     4626, Lr: 0.000300\n",
            "2020-05-19 11:46:27,813 Epoch   4 Step:     7700 Batch Loss:     2.005164 Tokens per Sec:     4564, Lr: 0.000300\n",
            "2020-05-19 11:47:16,479 Epoch   4 Step:     7800 Batch Loss:     1.855141 Tokens per Sec:     4671, Lr: 0.000300\n",
            "2020-05-19 11:48:05,143 Epoch   4 Step:     7900 Batch Loss:     1.991622 Tokens per Sec:     4737, Lr: 0.000300\n",
            "2020-05-19 11:48:53,893 Epoch   4 Step:     8000 Batch Loss:     2.059801 Tokens per Sec:     4623, Lr: 0.000300\n",
            "2020-05-19 11:50:21,146 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:50:21,147 Saving new checkpoint.\n",
            "2020-05-19 11:50:21,625 Example #0\n",
            "2020-05-19 11:50:21,625 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:50:21,626 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:50:21,626 \tHypothesis: Yehova o ṋea Yesu vhuṱolo .\n",
            "2020-05-19 11:50:21,626 Example #1\n",
            "2020-05-19 11:50:21,626 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:50:21,626 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:50:21,626 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi o vha a sa athu u fhisea hawe kana u sa fhelela hawe . — 2 Vha - Kor .\n",
            "2020-05-19 11:50:21,626 Example #2\n",
            "2020-05-19 11:50:21,627 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:50:21,627 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:50:21,627 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Timotheo uri a kone u ṱhogomela ṱhoḓea ya Paulo ya lwa ṋama .\n",
            "2020-05-19 11:50:21,627 Example #3\n",
            "2020-05-19 11:50:21,627 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:50:21,627 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:50:21,627 \tHypothesis: “ Ni ḓo tshimbila hani ’ nga ṅwambo wa uri ni ḓo tshimbila nga nḓila yo fhamba - fhambanaho ? — 1 VHAKORINTA 18 : 21 .\n",
            "2020-05-19 11:50:21,628 Validation result (greedy) at epoch   4, step     8000: bleu:  25.57, loss: 43076.9102, ppl:   6.3431, duration: 87.7345s\n",
            "2020-05-19 11:51:09,751 Epoch   4 Step:     8100 Batch Loss:     1.822089 Tokens per Sec:     4453, Lr: 0.000300\n",
            "2020-05-19 11:51:58,373 Epoch   4 Step:     8200 Batch Loss:     1.914015 Tokens per Sec:     4607, Lr: 0.000300\n",
            "2020-05-19 11:52:46,854 Epoch   4 Step:     8300 Batch Loss:     1.895945 Tokens per Sec:     4694, Lr: 0.000300\n",
            "2020-05-19 11:53:35,027 Epoch   4 Step:     8400 Batch Loss:     2.072012 Tokens per Sec:     4660, Lr: 0.000300\n",
            "2020-05-19 11:53:45,275 Epoch   4: total training loss 4451.92\n",
            "2020-05-19 11:53:45,275 EPOCH 5\n",
            "2020-05-19 11:54:23,568 Epoch   5 Step:     8500 Batch Loss:     1.940486 Tokens per Sec:     4587, Lr: 0.000300\n",
            "2020-05-19 11:55:12,302 Epoch   5 Step:     8600 Batch Loss:     1.812154 Tokens per Sec:     4568, Lr: 0.000300\n",
            "2020-05-19 11:56:00,864 Epoch   5 Step:     8700 Batch Loss:     2.009825 Tokens per Sec:     4641, Lr: 0.000300\n",
            "2020-05-19 11:56:49,471 Epoch   5 Step:     8800 Batch Loss:     1.800712 Tokens per Sec:     4626, Lr: 0.000300\n",
            "2020-05-19 11:57:38,000 Epoch   5 Step:     8900 Batch Loss:     1.972462 Tokens per Sec:     4593, Lr: 0.000300\n",
            "2020-05-19 11:58:26,916 Epoch   5 Step:     9000 Batch Loss:     1.979865 Tokens per Sec:     4643, Lr: 0.000300\n",
            "2020-05-19 11:59:42,476 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 11:59:42,477 Saving new checkpoint.\n",
            "2020-05-19 11:59:42,958 Example #0\n",
            "2020-05-19 11:59:42,958 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 11:59:42,958 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 11:59:42,958 \tHypothesis: Yehova o ṋea Yesu vhuṱolo .\n",
            "2020-05-19 11:59:42,958 Example #1\n",
            "2020-05-19 11:59:42,959 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 11:59:42,959 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 11:59:42,959 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a si tsha fhisetshela u fhisea kana u vha na mafulufulu . — 2 Vha - Kor .\n",
            "2020-05-19 11:59:42,959 Example #2\n",
            "2020-05-19 11:59:42,959 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 11:59:42,959 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 11:59:42,959 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Timotheo uri a ṱhogomele ṱhoḓea dza Paulo .\n",
            "2020-05-19 11:59:42,960 Example #3\n",
            "2020-05-19 11:59:42,960 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 11:59:42,960 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 11:59:42,960 \tHypothesis: “ Ni ḓo dzula ni tshi khou tshimbila hani kha mahumbulele mavhili ? ” — 1 VHAKORINTA 18 : 21 .\n",
            "2020-05-19 11:59:42,960 Validation result (greedy) at epoch   5, step     9000: bleu:  27.02, loss: 41350.0078, ppl:   5.8903, duration: 76.0440s\n",
            "2020-05-19 12:00:31,515 Epoch   5 Step:     9100 Batch Loss:     2.139071 Tokens per Sec:     4672, Lr: 0.000300\n",
            "2020-05-19 12:01:19,892 Epoch   5 Step:     9200 Batch Loss:     1.519234 Tokens per Sec:     4576, Lr: 0.000300\n",
            "2020-05-19 12:02:08,951 Epoch   5 Step:     9300 Batch Loss:     1.591758 Tokens per Sec:     4722, Lr: 0.000300\n",
            "2020-05-19 12:02:57,204 Epoch   5 Step:     9400 Batch Loss:     1.646113 Tokens per Sec:     4547, Lr: 0.000300\n",
            "2020-05-19 12:03:45,366 Epoch   5 Step:     9500 Batch Loss:     2.018197 Tokens per Sec:     4619, Lr: 0.000300\n",
            "2020-05-19 12:04:33,414 Epoch   5 Step:     9600 Batch Loss:     1.872828 Tokens per Sec:     4682, Lr: 0.000300\n",
            "2020-05-19 12:05:21,948 Epoch   5 Step:     9700 Batch Loss:     1.859946 Tokens per Sec:     4647, Lr: 0.000300\n",
            "2020-05-19 12:06:10,632 Epoch   5 Step:     9800 Batch Loss:     1.670569 Tokens per Sec:     4455, Lr: 0.000300\n",
            "2020-05-19 12:06:58,692 Epoch   5 Step:     9900 Batch Loss:     1.926366 Tokens per Sec:     4726, Lr: 0.000300\n",
            "2020-05-19 12:07:47,080 Epoch   5 Step:    10000 Batch Loss:     1.782223 Tokens per Sec:     4598, Lr: 0.000300\n",
            "2020-05-19 12:08:53,217 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:08:53,217 Saving new checkpoint.\n",
            "2020-05-19 12:08:53,671 Example #0\n",
            "2020-05-19 12:08:53,671 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:08:53,671 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:08:53,671 \tHypothesis: Yehova o ṋea Yesu vhuṱolo .\n",
            "2020-05-19 12:08:53,671 Example #1\n",
            "2020-05-19 12:08:53,671 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:08:53,672 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:08:53,672 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa athu u fhisetshela hawe kana u sa fhelela hawe . — 2 Vha - Kor .\n",
            "2020-05-19 12:08:53,672 Example #2\n",
            "2020-05-19 12:08:53,672 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:08:53,672 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:08:53,672 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela ngei Roma uri a ṱhogomele ṱhoḓea dza Paulo .\n",
            "2020-05-19 12:08:53,672 Example #3\n",
            "2020-05-19 12:08:53,673 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:08:53,673 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:08:53,673 \tHypothesis: “ Hu ḓo vha na tshifhinga tshingafhani ni tshi khou bvela phanḓa ni tshi shumisa mihumbulo mivhili mivhili mivhili i fanaho na ya u sa vha na mavhonele mavhili ? ” — 1 VHAKORINTA 18 : 21 .\n",
            "2020-05-19 12:08:53,673 Validation result (greedy) at epoch   5, step    10000: bleu:  28.77, loss: 39892.4688, ppl:   5.5334, duration: 66.5927s\n",
            "2020-05-19 12:09:41,992 Epoch   5 Step:    10100 Batch Loss:     1.993956 Tokens per Sec:     4544, Lr: 0.000300\n",
            "2020-05-19 12:10:29,789 Epoch   5 Step:    10200 Batch Loss:     1.787491 Tokens per Sec:     4590, Lr: 0.000300\n",
            "2020-05-19 12:11:18,031 Epoch   5 Step:    10300 Batch Loss:     1.927016 Tokens per Sec:     4673, Lr: 0.000300\n",
            "2020-05-19 12:12:06,452 Epoch   5 Step:    10400 Batch Loss:     1.588067 Tokens per Sec:     4657, Lr: 0.000300\n",
            "2020-05-19 12:12:55,144 Epoch   5 Step:    10500 Batch Loss:     1.850418 Tokens per Sec:     4635, Lr: 0.000300\n",
            "2020-05-19 12:13:06,959 Epoch   5: total training loss 4024.80\n",
            "2020-05-19 12:13:06,959 EPOCH 6\n",
            "2020-05-19 12:13:43,863 Epoch   6 Step:    10600 Batch Loss:     1.828255 Tokens per Sec:     4589, Lr: 0.000300\n",
            "2020-05-19 12:14:32,172 Epoch   6 Step:    10700 Batch Loss:     1.728193 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 12:15:21,012 Epoch   6 Step:    10800 Batch Loss:     1.530628 Tokens per Sec:     4731, Lr: 0.000300\n",
            "2020-05-19 12:16:09,245 Epoch   6 Step:    10900 Batch Loss:     2.013975 Tokens per Sec:     4726, Lr: 0.000300\n",
            "2020-05-19 12:16:57,744 Epoch   6 Step:    11000 Batch Loss:     1.599768 Tokens per Sec:     4644, Lr: 0.000300\n",
            "2020-05-19 12:18:11,856 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:18:11,856 Saving new checkpoint.\n",
            "2020-05-19 12:18:12,298 Example #0\n",
            "2020-05-19 12:18:12,298 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:18:12,298 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:18:12,298 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 12:18:12,299 Example #1\n",
            "2020-05-19 12:18:12,299 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:18:12,299 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:18:12,299 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa athu u fhisetshela hawe kana u sa fhelela hawe . — 2 Vha - Kor .\n",
            "2020-05-19 12:18:12,299 Example #2\n",
            "2020-05-19 12:18:12,299 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:18:12,300 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:18:12,300 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Vhakorinta uri a ṱhogomele ṱhoḓea ya Paulo ya ṋama .\n",
            "2020-05-19 12:18:12,300 Example #3\n",
            "2020-05-19 12:18:12,300 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:18:12,300 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:18:12,300 \tHypothesis: “ Ni ḓo dzula hani ni tshi khou bvela phanḓa ni tshi khou tshimbila kha mahumbulele mavhili ? ” — 1 VHA - KORINTA 18 : 21 .\n",
            "2020-05-19 12:18:12,300 Validation result (greedy) at epoch   6, step    11000: bleu:  29.93, loss: 38838.5977, ppl:   5.2889, duration: 74.5555s\n",
            "2020-05-19 12:19:00,915 Epoch   6 Step:    11100 Batch Loss:     1.803293 Tokens per Sec:     4694, Lr: 0.000300\n",
            "2020-05-19 12:19:48,409 Epoch   6 Step:    11200 Batch Loss:     1.681359 Tokens per Sec:     4604, Lr: 0.000300\n",
            "2020-05-19 12:20:36,527 Epoch   6 Step:    11300 Batch Loss:     1.597592 Tokens per Sec:     4571, Lr: 0.000300\n",
            "2020-05-19 12:21:24,660 Epoch   6 Step:    11400 Batch Loss:     1.881657 Tokens per Sec:     4648, Lr: 0.000300\n",
            "2020-05-19 12:22:13,016 Epoch   6 Step:    11500 Batch Loss:     1.589600 Tokens per Sec:     4539, Lr: 0.000300\n",
            "2020-05-19 12:23:01,316 Epoch   6 Step:    11600 Batch Loss:     1.624256 Tokens per Sec:     4644, Lr: 0.000300\n",
            "2020-05-19 12:23:49,590 Epoch   6 Step:    11700 Batch Loss:     1.565749 Tokens per Sec:     4656, Lr: 0.000300\n",
            "2020-05-19 12:24:37,884 Epoch   6 Step:    11800 Batch Loss:     1.554887 Tokens per Sec:     4585, Lr: 0.000300\n",
            "2020-05-19 12:25:26,201 Epoch   6 Step:    11900 Batch Loss:     1.689201 Tokens per Sec:     4696, Lr: 0.000300\n",
            "2020-05-19 12:26:13,589 Epoch   6 Step:    12000 Batch Loss:     1.726018 Tokens per Sec:     4636, Lr: 0.000300\n",
            "2020-05-19 12:27:27,099 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:27:27,099 Saving new checkpoint.\n",
            "2020-05-19 12:27:27,599 Example #0\n",
            "2020-05-19 12:27:27,599 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:27:27,599 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:27:27,599 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 12:27:27,599 Example #1\n",
            "2020-05-19 12:27:27,599 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:27:27,600 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:27:27,600 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wawe wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:27:27,600 Example #2\n",
            "2020-05-19 12:27:27,600 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:27:27,600 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:27:27,600 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Timotheo uri a ṱhogomele ṱhoḓea ya Paulo ya ṋama .\n",
            "2020-05-19 12:27:27,600 Example #3\n",
            "2020-05-19 12:27:27,600 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:27:27,601 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:27:27,601 \tHypothesis: “ Ni ḓo bvela phanḓa ni tshi khou bvela phanḓa ni tshi shumisa mihumbulo mivhili mivhili mivhili . ” — 1 VHA - KORINTA 18 : 21 .\n",
            "2020-05-19 12:27:27,601 Validation result (greedy) at epoch   6, step    12000: bleu:  30.90, loss: 37755.3086, ppl:   5.0488, duration: 74.0115s\n",
            "2020-05-19 12:28:16,202 Epoch   6 Step:    12100 Batch Loss:     2.482217 Tokens per Sec:     4691, Lr: 0.000300\n",
            "2020-05-19 12:29:04,972 Epoch   6 Step:    12200 Batch Loss:     1.776649 Tokens per Sec:     4693, Lr: 0.000300\n",
            "2020-05-19 12:29:52,700 Epoch   6 Step:    12300 Batch Loss:     1.594723 Tokens per Sec:     4575, Lr: 0.000300\n",
            "2020-05-19 12:30:41,508 Epoch   6 Step:    12400 Batch Loss:     2.083701 Tokens per Sec:     4592, Lr: 0.000300\n",
            "2020-05-19 12:31:29,742 Epoch   6 Step:    12500 Batch Loss:     1.608701 Tokens per Sec:     4672, Lr: 0.000300\n",
            "2020-05-19 12:32:18,471 Epoch   6 Step:    12600 Batch Loss:     1.930710 Tokens per Sec:     4622, Lr: 0.000300\n",
            "2020-05-19 12:32:31,713 Epoch   6: total training loss 3754.62\n",
            "2020-05-19 12:32:31,713 EPOCH 7\n",
            "2020-05-19 12:33:06,420 Epoch   7 Step:    12700 Batch Loss:     1.754573 Tokens per Sec:     4568, Lr: 0.000300\n",
            "2020-05-19 12:33:54,477 Epoch   7 Step:    12800 Batch Loss:     1.720581 Tokens per Sec:     4637, Lr: 0.000300\n",
            "2020-05-19 12:34:42,550 Epoch   7 Step:    12900 Batch Loss:     1.561592 Tokens per Sec:     4573, Lr: 0.000300\n",
            "2020-05-19 12:35:30,296 Epoch   7 Step:    13000 Batch Loss:     1.906374 Tokens per Sec:     4653, Lr: 0.000300\n",
            "2020-05-19 12:36:42,551 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:36:42,552 Saving new checkpoint.\n",
            "2020-05-19 12:36:43,027 Example #0\n",
            "2020-05-19 12:36:43,027 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:36:43,028 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:36:43,028 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 12:36:43,028 Example #1\n",
            "2020-05-19 12:36:43,028 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:36:43,028 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:36:43,028 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a xelelwa nga u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:36:43,028 Example #2\n",
            "2020-05-19 12:36:43,029 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:36:43,029 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:36:43,029 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ṱhogomele ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 12:36:43,029 Example #3\n",
            "2020-05-19 12:36:43,029 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:36:43,029 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:36:43,029 \tHypothesis: “ Ni ḓo bvela phanḓa ni tshi litsha u fhaṱa mihumbulo mivhili ” — 1 VHA - TSHI TETESAHO : 18 .\n",
            "2020-05-19 12:36:43,029 Validation result (greedy) at epoch   7, step    13000: bleu:  32.15, loss: 37271.9102, ppl:   4.9452, duration: 72.7326s\n",
            "2020-05-19 12:37:31,376 Epoch   7 Step:    13100 Batch Loss:     1.626494 Tokens per Sec:     4665, Lr: 0.000300\n",
            "2020-05-19 12:38:19,704 Epoch   7 Step:    13200 Batch Loss:     2.117931 Tokens per Sec:     4636, Lr: 0.000300\n",
            "2020-05-19 12:39:08,254 Epoch   7 Step:    13300 Batch Loss:     1.343230 Tokens per Sec:     4634, Lr: 0.000300\n",
            "2020-05-19 12:39:56,539 Epoch   7 Step:    13400 Batch Loss:     1.903207 Tokens per Sec:     4683, Lr: 0.000300\n",
            "2020-05-19 12:40:45,230 Epoch   7 Step:    13500 Batch Loss:     1.808798 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 12:41:33,288 Epoch   7 Step:    13600 Batch Loss:     1.876101 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 12:42:21,456 Epoch   7 Step:    13700 Batch Loss:     1.405341 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 12:43:10,236 Epoch   7 Step:    13800 Batch Loss:     1.648861 Tokens per Sec:     4716, Lr: 0.000300\n",
            "2020-05-19 12:43:58,334 Epoch   7 Step:    13900 Batch Loss:     1.399321 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 12:44:46,538 Epoch   7 Step:    14000 Batch Loss:     1.498226 Tokens per Sec:     4536, Lr: 0.000300\n",
            "2020-05-19 12:46:09,124 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:46:09,124 Saving new checkpoint.\n",
            "2020-05-19 12:46:09,626 Example #0\n",
            "2020-05-19 12:46:09,626 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:46:09,627 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:46:09,627 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 12:46:09,627 Example #1\n",
            "2020-05-19 12:46:09,627 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:46:09,627 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:46:09,627 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi o vha a sa athu u fhisetshela u fhisea kana u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 12:46:09,627 Example #2\n",
            "2020-05-19 12:46:09,628 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:46:09,628 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:46:09,628 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ṱhogomele ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 12:46:09,628 Example #3\n",
            "2020-05-19 12:46:09,628 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:46:09,628 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:46:09,628 \tHypothesis: “ Ni ḓo dzula hani ni tshi khou bvela phanḓa ni tshi litsha u dzula ni tshi humbula nga ha mavhonele mavhili ? ” — 1 VHAKORINTA 18 : 21 .\n",
            "2020-05-19 12:46:09,629 Validation result (greedy) at epoch   7, step    14000: bleu:  32.01, loss: 36745.8164, ppl:   4.8349, duration: 83.0899s\n",
            "2020-05-19 12:46:57,384 Epoch   7 Step:    14100 Batch Loss:     1.556270 Tokens per Sec:     4662, Lr: 0.000300\n",
            "2020-05-19 12:47:45,992 Epoch   7 Step:    14200 Batch Loss:     1.473203 Tokens per Sec:     4741, Lr: 0.000300\n",
            "2020-05-19 12:48:34,353 Epoch   7 Step:    14300 Batch Loss:     1.744795 Tokens per Sec:     4549, Lr: 0.000300\n",
            "2020-05-19 12:49:22,332 Epoch   7 Step:    14400 Batch Loss:     1.957216 Tokens per Sec:     4593, Lr: 0.000300\n",
            "2020-05-19 12:50:10,908 Epoch   7 Step:    14500 Batch Loss:     1.666852 Tokens per Sec:     4734, Lr: 0.000300\n",
            "2020-05-19 12:50:59,492 Epoch   7 Step:    14600 Batch Loss:     1.694974 Tokens per Sec:     4645, Lr: 0.000300\n",
            "2020-05-19 12:51:47,425 Epoch   7 Step:    14700 Batch Loss:     1.824747 Tokens per Sec:     4627, Lr: 0.000300\n",
            "2020-05-19 12:52:03,738 Epoch   7: total training loss 3556.71\n",
            "2020-05-19 12:52:03,738 EPOCH 8\n",
            "2020-05-19 12:52:36,302 Epoch   8 Step:    14800 Batch Loss:     1.597796 Tokens per Sec:     4613, Lr: 0.000300\n",
            "2020-05-19 12:53:24,688 Epoch   8 Step:    14900 Batch Loss:     1.507406 Tokens per Sec:     4507, Lr: 0.000300\n",
            "2020-05-19 12:54:13,522 Epoch   8 Step:    15000 Batch Loss:     1.641913 Tokens per Sec:     4642, Lr: 0.000300\n",
            "2020-05-19 12:55:11,723 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 12:55:11,723 Saving new checkpoint.\n",
            "2020-05-19 12:55:12,203 Example #0\n",
            "2020-05-19 12:55:12,203 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 12:55:12,203 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 12:55:12,203 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 12:55:12,204 Example #1\n",
            "2020-05-19 12:55:12,204 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 12:55:12,204 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:55:12,204 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u ṱavhanyiswa ha u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 12:55:12,204 Example #2\n",
            "2020-05-19 12:55:12,205 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 12:55:12,205 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 12:55:12,205 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma uri a ṱhogomele ṱhoḓea dza Paulo .\n",
            "2020-05-19 12:55:12,205 Example #3\n",
            "2020-05-19 12:55:12,205 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 12:55:12,205 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 12:55:12,206 \tHypothesis: “ Ni ḓo dzula hani ni tshi khou bvela phanḓa ni tshi livhisa mihumbulo mivhili ? ” — 1 VHA - KORINTA 18 : 21 .\n",
            "2020-05-19 12:55:12,206 Validation result (greedy) at epoch   8, step    15000: bleu:  33.47, loss: 35634.1133, ppl:   4.6098, duration: 58.6828s\n",
            "2020-05-19 12:56:01,209 Epoch   8 Step:    15100 Batch Loss:     1.803761 Tokens per Sec:     4651, Lr: 0.000300\n",
            "2020-05-19 12:56:49,575 Epoch   8 Step:    15200 Batch Loss:     1.765244 Tokens per Sec:     4580, Lr: 0.000300\n",
            "2020-05-19 12:57:38,277 Epoch   8 Step:    15300 Batch Loss:     1.574148 Tokens per Sec:     4700, Lr: 0.000300\n",
            "2020-05-19 12:58:26,840 Epoch   8 Step:    15400 Batch Loss:     1.767392 Tokens per Sec:     4579, Lr: 0.000300\n",
            "2020-05-19 12:59:15,515 Epoch   8 Step:    15500 Batch Loss:     1.268235 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 13:00:03,921 Epoch   8 Step:    15600 Batch Loss:     2.268845 Tokens per Sec:     4635, Lr: 0.000300\n",
            "2020-05-19 13:00:52,634 Epoch   8 Step:    15700 Batch Loss:     1.909771 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 13:01:41,308 Epoch   8 Step:    15800 Batch Loss:     1.519077 Tokens per Sec:     4671, Lr: 0.000300\n",
            "2020-05-19 13:02:29,607 Epoch   8 Step:    15900 Batch Loss:     1.453497 Tokens per Sec:     4563, Lr: 0.000300\n",
            "2020-05-19 13:03:18,466 Epoch   8 Step:    16000 Batch Loss:     1.606412 Tokens per Sec:     4616, Lr: 0.000300\n",
            "2020-05-19 13:04:39,401 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:04:39,401 Saving new checkpoint.\n",
            "2020-05-19 13:04:39,907 Example #0\n",
            "2020-05-19 13:04:39,908 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:04:39,908 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:04:39,908 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:04:39,908 Example #1\n",
            "2020-05-19 13:04:39,908 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:04:39,908 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:04:39,909 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na mavhonele a u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 13:04:39,909 Example #2\n",
            "2020-05-19 13:04:39,909 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:04:39,909 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:04:39,909 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:04:39,910 Example #3\n",
            "2020-05-19 13:04:39,910 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:04:39,910 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:04:39,910 \tHypothesis: “ Ni ḓo dzula ni tshi khou bvela phanḓa ni tshi khou bvela phanḓa ni tshi khou bvela phanḓa ni tshi khou rabela zwithu zwivhili ? ” — 1 VHA - KORINTA 18 : 21 .\n",
            "2020-05-19 13:04:39,910 Validation result (greedy) at epoch   8, step    16000: bleu:  34.15, loss: 35063.9336, ppl:   4.4984, duration: 81.4440s\n",
            "2020-05-19 13:05:27,718 Epoch   8 Step:    16100 Batch Loss:     1.450926 Tokens per Sec:     4620, Lr: 0.000300\n",
            "2020-05-19 13:06:16,275 Epoch   8 Step:    16200 Batch Loss:     1.701348 Tokens per Sec:     4556, Lr: 0.000300\n",
            "2020-05-19 13:07:04,495 Epoch   8 Step:    16300 Batch Loss:     1.645967 Tokens per Sec:     4497, Lr: 0.000300\n",
            "2020-05-19 13:07:53,288 Epoch   8 Step:    16400 Batch Loss:     1.392908 Tokens per Sec:     4616, Lr: 0.000300\n",
            "2020-05-19 13:08:41,504 Epoch   8 Step:    16500 Batch Loss:     1.555173 Tokens per Sec:     4710, Lr: 0.000300\n",
            "2020-05-19 13:09:29,720 Epoch   8 Step:    16600 Batch Loss:     1.416193 Tokens per Sec:     4604, Lr: 0.000300\n",
            "2020-05-19 13:10:17,643 Epoch   8 Step:    16700 Batch Loss:     1.569597 Tokens per Sec:     4541, Lr: 0.000300\n",
            "2020-05-19 13:11:06,394 Epoch   8 Step:    16800 Batch Loss:     1.702362 Tokens per Sec:     4672, Lr: 0.000300\n",
            "2020-05-19 13:11:26,500 Epoch   8: total training loss 3399.38\n",
            "2020-05-19 13:11:26,500 EPOCH 9\n",
            "2020-05-19 13:11:54,758 Epoch   9 Step:    16900 Batch Loss:     1.194049 Tokens per Sec:     4467, Lr: 0.000300\n",
            "2020-05-19 13:12:43,723 Epoch   9 Step:    17000 Batch Loss:     1.739185 Tokens per Sec:     4600, Lr: 0.000300\n",
            "2020-05-19 13:13:46,182 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:13:46,182 Saving new checkpoint.\n",
            "2020-05-19 13:13:46,655 Example #0\n",
            "2020-05-19 13:13:46,656 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:13:46,656 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:13:46,656 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:13:46,656 Example #1\n",
            "2020-05-19 13:13:46,656 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:13:46,656 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:13:46,657 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u ṱavhanyiswa ha u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 13:13:46,657 Example #2\n",
            "2020-05-19 13:13:46,657 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:13:46,657 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:13:46,657 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo .\n",
            "2020-05-19 13:13:46,657 Example #3\n",
            "2020-05-19 13:13:46,657 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:13:46,658 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:13:46,658 \tHypothesis: “ Ni ḓo litsha hani u ṱudza thungo dzoṱhe mihumbulo mivhili ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 13:13:46,658 Validation result (greedy) at epoch   9, step    17000: bleu:  34.14, loss: 34512.6680, ppl:   4.3933, duration: 62.9342s\n",
            "2020-05-19 13:14:35,223 Epoch   9 Step:    17100 Batch Loss:     1.546732 Tokens per Sec:     4646, Lr: 0.000300\n",
            "2020-05-19 13:15:23,521 Epoch   9 Step:    17200 Batch Loss:     1.339845 Tokens per Sec:     4562, Lr: 0.000300\n",
            "2020-05-19 13:16:11,860 Epoch   9 Step:    17300 Batch Loss:     1.595270 Tokens per Sec:     4573, Lr: 0.000300\n",
            "2020-05-19 13:17:00,455 Epoch   9 Step:    17400 Batch Loss:     1.273439 Tokens per Sec:     4593, Lr: 0.000300\n",
            "2020-05-19 13:17:49,242 Epoch   9 Step:    17500 Batch Loss:     1.430919 Tokens per Sec:     4617, Lr: 0.000300\n",
            "2020-05-19 13:18:37,823 Epoch   9 Step:    17600 Batch Loss:     1.435511 Tokens per Sec:     4544, Lr: 0.000300\n",
            "2020-05-19 13:19:26,423 Epoch   9 Step:    17700 Batch Loss:     1.404823 Tokens per Sec:     4665, Lr: 0.000300\n",
            "2020-05-19 13:20:14,582 Epoch   9 Step:    17800 Batch Loss:     1.505700 Tokens per Sec:     4581, Lr: 0.000300\n",
            "2020-05-19 13:21:03,349 Epoch   9 Step:    17900 Batch Loss:     1.362078 Tokens per Sec:     4630, Lr: 0.000300\n",
            "2020-05-19 13:21:51,870 Epoch   9 Step:    18000 Batch Loss:     1.347209 Tokens per Sec:     4670, Lr: 0.000300\n",
            "2020-05-19 13:23:11,957 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:23:11,957 Saving new checkpoint.\n",
            "2020-05-19 13:23:12,552 Example #0\n",
            "2020-05-19 13:23:12,552 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:23:12,552 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:23:12,553 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:23:12,553 Example #1\n",
            "2020-05-19 13:23:12,553 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:23:12,553 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:23:12,553 \tHypothesis: O konḓelela vhuleme vhunzhi , naho ho ngo vhuya a litsha u fhisea hawe kana u ṱavhanyiswa ha u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:23:12,553 Example #2\n",
            "2020-05-19 13:23:12,553 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:23:12,554 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:23:12,554 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:23:12,554 Example #3\n",
            "2020-05-19 13:23:12,554 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:23:12,554 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:23:12,554 \tHypothesis: “ Ni ḓo dzula ni tshi rabela misi yoṱhe , ni ḓo litsha u dzula ni tshi rabela nga nḓila - ḓe ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 13:23:12,554 Validation result (greedy) at epoch   9, step    18000: bleu:  34.61, loss: 34134.8711, ppl:   4.3227, duration: 80.6835s\n",
            "2020-05-19 13:24:01,099 Epoch   9 Step:    18100 Batch Loss:     1.511880 Tokens per Sec:     4635, Lr: 0.000300\n",
            "2020-05-19 13:24:50,102 Epoch   9 Step:    18200 Batch Loss:     1.437485 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 13:25:39,015 Epoch   9 Step:    18300 Batch Loss:     1.743662 Tokens per Sec:     4549, Lr: 0.000300\n",
            "2020-05-19 13:26:27,358 Epoch   9 Step:    18400 Batch Loss:     1.570728 Tokens per Sec:     4598, Lr: 0.000300\n",
            "2020-05-19 13:27:15,451 Epoch   9 Step:    18500 Batch Loss:     1.589478 Tokens per Sec:     4624, Lr: 0.000300\n",
            "2020-05-19 13:28:04,078 Epoch   9 Step:    18600 Batch Loss:     1.447941 Tokens per Sec:     4585, Lr: 0.000300\n",
            "2020-05-19 13:28:52,725 Epoch   9 Step:    18700 Batch Loss:     1.646245 Tokens per Sec:     4692, Lr: 0.000300\n",
            "2020-05-19 13:29:41,515 Epoch   9 Step:    18800 Batch Loss:     2.060736 Tokens per Sec:     4642, Lr: 0.000300\n",
            "2020-05-19 13:30:30,074 Epoch   9 Step:    18900 Batch Loss:     1.320987 Tokens per Sec:     4569, Lr: 0.000300\n",
            "2020-05-19 13:30:52,493 Epoch   9: total training loss 3254.58\n",
            "2020-05-19 13:30:52,493 EPOCH 10\n",
            "2020-05-19 13:31:18,545 Epoch  10 Step:    19000 Batch Loss:     1.373727 Tokens per Sec:     4761, Lr: 0.000300\n",
            "2020-05-19 13:32:18,644 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:32:18,644 Saving new checkpoint.\n",
            "2020-05-19 13:32:19,112 Example #0\n",
            "2020-05-19 13:32:19,112 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:32:19,113 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:32:19,113 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:32:19,113 Example #1\n",
            "2020-05-19 13:32:19,113 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:32:19,113 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:32:19,113 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 13:32:19,113 Example #2\n",
            "2020-05-19 13:32:19,115 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:32:19,115 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:32:19,115 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:32:19,115 Example #3\n",
            "2020-05-19 13:32:19,115 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:32:19,115 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:32:19,115 \tHypothesis: “ Ni ḓo dzula hani ni tshi khou ṱudza thungo dzoṱhe , ni ḓo litsha u dzula ni tshi humbula nga ha mafhungo mavhili ? ” — 1 VHA - ROMA 18 : 21 .\n",
            "2020-05-19 13:32:19,116 Validation result (greedy) at epoch  10, step    19000: bleu:  35.39, loss: 33658.5117, ppl:   4.2353, duration: 60.5698s\n",
            "2020-05-19 13:33:07,327 Epoch  10 Step:    19100 Batch Loss:     2.076158 Tokens per Sec:     4622, Lr: 0.000300\n",
            "2020-05-19 13:33:55,710 Epoch  10 Step:    19200 Batch Loss:     1.581242 Tokens per Sec:     4597, Lr: 0.000300\n",
            "2020-05-19 13:34:43,894 Epoch  10 Step:    19300 Batch Loss:     1.343452 Tokens per Sec:     4603, Lr: 0.000300\n",
            "2020-05-19 13:35:32,272 Epoch  10 Step:    19400 Batch Loss:     1.229649 Tokens per Sec:     4586, Lr: 0.000300\n",
            "2020-05-19 13:36:21,111 Epoch  10 Step:    19500 Batch Loss:     1.277190 Tokens per Sec:     4657, Lr: 0.000300\n",
            "2020-05-19 13:37:09,501 Epoch  10 Step:    19600 Batch Loss:     1.678316 Tokens per Sec:     4593, Lr: 0.000300\n",
            "2020-05-19 13:37:58,432 Epoch  10 Step:    19700 Batch Loss:     1.723859 Tokens per Sec:     4719, Lr: 0.000300\n",
            "2020-05-19 13:38:47,419 Epoch  10 Step:    19800 Batch Loss:     1.743610 Tokens per Sec:     4617, Lr: 0.000300\n",
            "2020-05-19 13:39:36,479 Epoch  10 Step:    19900 Batch Loss:     1.995142 Tokens per Sec:     4696, Lr: 0.000300\n",
            "2020-05-19 13:40:25,074 Epoch  10 Step:    20000 Batch Loss:     1.459509 Tokens per Sec:     4636, Lr: 0.000300\n",
            "2020-05-19 13:41:30,557 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:41:30,557 Saving new checkpoint.\n",
            "2020-05-19 13:41:31,043 Example #0\n",
            "2020-05-19 13:41:31,043 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:41:31,043 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:41:31,043 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:41:31,043 Example #1\n",
            "2020-05-19 13:41:31,044 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:41:31,044 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:41:31,044 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:41:31,044 Example #2\n",
            "2020-05-19 13:41:31,044 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:41:31,044 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:41:31,045 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:41:31,045 Example #3\n",
            "2020-05-19 13:41:31,046 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:41:31,046 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:41:31,046 \tHypothesis: “ Ni ḓo dzula ni tshi rabela misi yoṱhe , ni ḓo litsha u dzula ni tshi humbula nga ha mavhonele mavhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 13:41:31,046 Validation result (greedy) at epoch  10, step    20000: bleu:  35.77, loss: 33328.2188, ppl:   4.1757, duration: 65.9715s\n",
            "2020-05-19 13:42:19,453 Epoch  10 Step:    20100 Batch Loss:     1.313514 Tokens per Sec:     4611, Lr: 0.000300\n",
            "2020-05-19 13:43:07,744 Epoch  10 Step:    20200 Batch Loss:     1.901368 Tokens per Sec:     4590, Lr: 0.000300\n",
            "2020-05-19 13:43:55,787 Epoch  10 Step:    20300 Batch Loss:     1.594694 Tokens per Sec:     4615, Lr: 0.000300\n",
            "2020-05-19 13:44:44,567 Epoch  10 Step:    20400 Batch Loss:     1.460039 Tokens per Sec:     4581, Lr: 0.000300\n",
            "2020-05-19 13:45:33,236 Epoch  10 Step:    20500 Batch Loss:     1.432166 Tokens per Sec:     4619, Lr: 0.000300\n",
            "2020-05-19 13:46:22,055 Epoch  10 Step:    20600 Batch Loss:     1.710840 Tokens per Sec:     4652, Lr: 0.000300\n",
            "2020-05-19 13:47:10,369 Epoch  10 Step:    20700 Batch Loss:     1.742506 Tokens per Sec:     4604, Lr: 0.000300\n",
            "2020-05-19 13:47:59,106 Epoch  10 Step:    20800 Batch Loss:     1.113191 Tokens per Sec:     4659, Lr: 0.000300\n",
            "2020-05-19 13:48:47,000 Epoch  10 Step:    20900 Batch Loss:     1.391182 Tokens per Sec:     4500, Lr: 0.000300\n",
            "2020-05-19 13:49:35,628 Epoch  10 Step:    21000 Batch Loss:     1.624355 Tokens per Sec:     4575, Lr: 0.000300\n",
            "2020-05-19 13:50:41,552 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:50:41,552 Saving new checkpoint.\n",
            "2020-05-19 13:50:42,048 Example #0\n",
            "2020-05-19 13:50:42,048 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:50:42,049 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:50:42,049 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:50:42,049 Example #1\n",
            "2020-05-19 13:50:42,049 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:50:42,049 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:50:42,049 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ha athu vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:50:42,049 Example #2\n",
            "2020-05-19 13:50:42,050 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:50:42,050 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:50:42,050 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:50:42,050 Example #3\n",
            "2020-05-19 13:50:42,050 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:50:42,050 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:50:42,050 \tHypothesis: “ Ni ḓo litsha hani u ṱudza thungo dzoṱhe mihumbulo mivhili yo fhamba - fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 13:50:42,050 Validation result (greedy) at epoch  10, step    21000: bleu:  36.34, loss: 33101.5000, ppl:   4.1353, duration: 66.4222s\n",
            "2020-05-19 13:51:05,735 Epoch  10: total training loss 3145.17\n",
            "2020-05-19 13:51:05,736 EPOCH 11\n",
            "2020-05-19 13:51:30,360 Epoch  11 Step:    21100 Batch Loss:     1.690315 Tokens per Sec:     4577, Lr: 0.000300\n",
            "2020-05-19 13:52:18,723 Epoch  11 Step:    21200 Batch Loss:     1.478229 Tokens per Sec:     4692, Lr: 0.000300\n",
            "2020-05-19 13:53:07,275 Epoch  11 Step:    21300 Batch Loss:     1.221741 Tokens per Sec:     4623, Lr: 0.000300\n",
            "2020-05-19 13:53:56,195 Epoch  11 Step:    21400 Batch Loss:     1.438457 Tokens per Sec:     4597, Lr: 0.000300\n",
            "2020-05-19 13:54:44,757 Epoch  11 Step:    21500 Batch Loss:     1.574977 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 13:55:32,752 Epoch  11 Step:    21600 Batch Loss:     1.221858 Tokens per Sec:     4585, Lr: 0.000300\n",
            "2020-05-19 13:56:21,239 Epoch  11 Step:    21700 Batch Loss:     1.417622 Tokens per Sec:     4549, Lr: 0.000300\n",
            "2020-05-19 13:57:10,145 Epoch  11 Step:    21800 Batch Loss:     1.354766 Tokens per Sec:     4670, Lr: 0.000300\n",
            "2020-05-19 13:57:58,804 Epoch  11 Step:    21900 Batch Loss:     1.641940 Tokens per Sec:     4589, Lr: 0.000300\n",
            "2020-05-19 13:58:47,360 Epoch  11 Step:    22000 Batch Loss:     1.462293 Tokens per Sec:     4601, Lr: 0.000300\n",
            "2020-05-19 13:59:48,400 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 13:59:48,400 Saving new checkpoint.\n",
            "2020-05-19 13:59:48,833 Example #0\n",
            "2020-05-19 13:59:48,834 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 13:59:48,834 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 13:59:48,834 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 13:59:48,834 Example #1\n",
            "2020-05-19 13:59:48,834 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 13:59:48,834 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 13:59:48,834 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa koni u fhisea kana u fhisetshela hawe u ṱavhanyiswa . — 2 Vha - Kor .\n",
            "2020-05-19 13:59:48,835 Example #2\n",
            "2020-05-19 13:59:48,835 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 13:59:48,835 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 13:59:48,835 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 13:59:48,835 Example #3\n",
            "2020-05-19 13:59:48,836 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 13:59:48,836 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 13:59:48,836 \tHypothesis: “ Ni ḓo dzula ni tshi litsha u dzula ni tshi humbula nga ha mavhonele mavhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 13:59:48,836 Validation result (greedy) at epoch  11, step    22000: bleu:  35.93, loss: 32817.6562, ppl:   4.0853, duration: 61.4754s\n",
            "2020-05-19 14:00:37,355 Epoch  11 Step:    22100 Batch Loss:     1.972189 Tokens per Sec:     4781, Lr: 0.000300\n",
            "2020-05-19 14:01:25,402 Epoch  11 Step:    22200 Batch Loss:     1.380894 Tokens per Sec:     4543, Lr: 0.000300\n",
            "2020-05-19 14:02:14,210 Epoch  11 Step:    22300 Batch Loss:     1.102972 Tokens per Sec:     4693, Lr: 0.000300\n",
            "2020-05-19 14:03:02,596 Epoch  11 Step:    22400 Batch Loss:     1.371613 Tokens per Sec:     4597, Lr: 0.000300\n",
            "2020-05-19 14:03:51,042 Epoch  11 Step:    22500 Batch Loss:     1.228444 Tokens per Sec:     4543, Lr: 0.000300\n",
            "2020-05-19 14:04:39,942 Epoch  11 Step:    22600 Batch Loss:     1.251690 Tokens per Sec:     4638, Lr: 0.000300\n",
            "2020-05-19 14:05:28,627 Epoch  11 Step:    22700 Batch Loss:     1.264421 Tokens per Sec:     4557, Lr: 0.000300\n",
            "2020-05-19 14:06:16,848 Epoch  11 Step:    22800 Batch Loss:     1.743218 Tokens per Sec:     4533, Lr: 0.000300\n",
            "2020-05-19 14:07:05,285 Epoch  11 Step:    22900 Batch Loss:     1.615245 Tokens per Sec:     4746, Lr: 0.000300\n",
            "2020-05-19 14:07:53,263 Epoch  11 Step:    23000 Batch Loss:     1.132008 Tokens per Sec:     4564, Lr: 0.000300\n",
            "2020-05-19 14:09:01,286 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:09:01,287 Saving new checkpoint.\n",
            "2020-05-19 14:09:01,716 Example #0\n",
            "2020-05-19 14:09:01,716 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:09:01,716 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:09:01,717 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:09:01,717 Example #1\n",
            "2020-05-19 14:09:01,717 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:09:01,717 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:09:01,717 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 14:09:01,717 Example #2\n",
            "2020-05-19 14:09:01,718 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:09:01,718 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:09:01,718 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:09:01,718 Example #3\n",
            "2020-05-19 14:09:01,718 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:09:01,718 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:09:01,718 \tHypothesis: “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe kha mihumbulo mivhili yo fhamba - fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:09:01,718 Validation result (greedy) at epoch  11, step    23000: bleu:  36.44, loss: 32373.0020, ppl:   4.0081, duration: 68.4550s\n",
            "2020-05-19 14:09:49,811 Epoch  11 Step:    23100 Batch Loss:     1.556453 Tokens per Sec:     4631, Lr: 0.000300\n",
            "2020-05-19 14:10:16,230 Epoch  11: total training loss 3062.69\n",
            "2020-05-19 14:10:16,230 EPOCH 12\n",
            "2020-05-19 14:10:38,463 Epoch  12 Step:    23200 Batch Loss:     1.465875 Tokens per Sec:     4514, Lr: 0.000300\n",
            "2020-05-19 14:11:27,297 Epoch  12 Step:    23300 Batch Loss:     1.525598 Tokens per Sec:     4620, Lr: 0.000300\n",
            "2020-05-19 14:12:15,646 Epoch  12 Step:    23400 Batch Loss:     1.379183 Tokens per Sec:     4627, Lr: 0.000300\n",
            "2020-05-19 14:13:04,097 Epoch  12 Step:    23500 Batch Loss:     1.340683 Tokens per Sec:     4654, Lr: 0.000300\n",
            "2020-05-19 14:13:52,356 Epoch  12 Step:    23600 Batch Loss:     1.627360 Tokens per Sec:     4664, Lr: 0.000300\n",
            "2020-05-19 14:14:41,139 Epoch  12 Step:    23700 Batch Loss:     1.353012 Tokens per Sec:     4620, Lr: 0.000300\n",
            "2020-05-19 14:15:29,827 Epoch  12 Step:    23800 Batch Loss:     1.396169 Tokens per Sec:     4629, Lr: 0.000300\n",
            "2020-05-19 14:16:18,342 Epoch  12 Step:    23900 Batch Loss:     1.397413 Tokens per Sec:     4656, Lr: 0.000300\n",
            "2020-05-19 14:17:06,909 Epoch  12 Step:    24000 Batch Loss:     1.404331 Tokens per Sec:     4720, Lr: 0.000300\n",
            "2020-05-19 14:18:13,351 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:18:13,351 Saving new checkpoint.\n",
            "2020-05-19 14:18:13,790 Example #0\n",
            "2020-05-19 14:18:13,790 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:18:13,790 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:18:13,791 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:18:13,791 Example #1\n",
            "2020-05-19 14:18:13,791 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:18:13,791 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:18:13,791 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa athu vhuya a fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:18:13,791 Example #2\n",
            "2020-05-19 14:18:13,791 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:18:13,791 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:18:13,792 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:18:13,792 Example #3\n",
            "2020-05-19 14:18:13,792 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:18:13,792 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:18:13,792 \tHypothesis: “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe kha mihumbulo mivhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:18:13,792 Validation result (greedy) at epoch  12, step    24000: bleu:  37.04, loss: 32040.8867, ppl:   3.9515, duration: 66.8826s\n",
            "2020-05-19 14:19:02,019 Epoch  12 Step:    24100 Batch Loss:     1.629221 Tokens per Sec:     4659, Lr: 0.000300\n",
            "2020-05-19 14:19:50,825 Epoch  12 Step:    24200 Batch Loss:     1.420111 Tokens per Sec:     4757, Lr: 0.000300\n",
            "2020-05-19 14:20:39,349 Epoch  12 Step:    24300 Batch Loss:     1.437793 Tokens per Sec:     4631, Lr: 0.000300\n",
            "2020-05-19 14:21:27,468 Epoch  12 Step:    24400 Batch Loss:     1.322745 Tokens per Sec:     4612, Lr: 0.000300\n",
            "2020-05-19 14:22:15,399 Epoch  12 Step:    24500 Batch Loss:     1.609095 Tokens per Sec:     4558, Lr: 0.000300\n",
            "2020-05-19 14:23:04,293 Epoch  12 Step:    24600 Batch Loss:     1.615564 Tokens per Sec:     4642, Lr: 0.000300\n",
            "2020-05-19 14:23:52,619 Epoch  12 Step:    24700 Batch Loss:     1.307963 Tokens per Sec:     4565, Lr: 0.000300\n",
            "2020-05-19 14:24:40,335 Epoch  12 Step:    24800 Batch Loss:     1.372704 Tokens per Sec:     4570, Lr: 0.000300\n",
            "2020-05-19 14:25:28,989 Epoch  12 Step:    24900 Batch Loss:     1.608707 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 14:26:16,977 Epoch  12 Step:    25000 Batch Loss:     1.386162 Tokens per Sec:     4569, Lr: 0.000300\n",
            "2020-05-19 14:27:14,266 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:27:14,266 Saving new checkpoint.\n",
            "2020-05-19 14:27:14,707 Example #0\n",
            "2020-05-19 14:27:14,708 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:27:14,708 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:27:14,708 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:27:14,708 Example #1\n",
            "2020-05-19 14:27:14,708 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:27:14,708 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:27:14,709 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:27:14,709 Example #2\n",
            "2020-05-19 14:27:14,709 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:27:14,709 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:27:14,709 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:27:14,709 Example #3\n",
            "2020-05-19 14:27:14,709 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:27:14,709 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:27:14,710 \tHypothesis: “ Ni ḓo ṱudza thungo dzoṱhe mihumbulo mivhili yo fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:27:14,710 Validation result (greedy) at epoch  12, step    25000: bleu:  37.15, loss: 31832.5273, ppl:   3.9163, duration: 57.7322s\n",
            "2020-05-19 14:28:03,092 Epoch  12 Step:    25100 Batch Loss:     1.436224 Tokens per Sec:     4539, Lr: 0.000300\n",
            "2020-05-19 14:28:51,459 Epoch  12 Step:    25200 Batch Loss:     1.418938 Tokens per Sec:     4722, Lr: 0.000300\n",
            "2020-05-19 14:29:18,123 Epoch  12: total training loss 2968.86\n",
            "2020-05-19 14:29:18,123 EPOCH 13\n",
            "2020-05-19 14:29:40,063 Epoch  13 Step:    25300 Batch Loss:     1.389039 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 14:30:28,546 Epoch  13 Step:    25400 Batch Loss:     1.200104 Tokens per Sec:     4618, Lr: 0.000300\n",
            "2020-05-19 14:31:17,402 Epoch  13 Step:    25500 Batch Loss:     1.518700 Tokens per Sec:     4683, Lr: 0.000300\n",
            "2020-05-19 14:32:05,830 Epoch  13 Step:    25600 Batch Loss:     1.757522 Tokens per Sec:     4652, Lr: 0.000300\n",
            "2020-05-19 14:32:53,703 Epoch  13 Step:    25700 Batch Loss:     1.521359 Tokens per Sec:     4652, Lr: 0.000300\n",
            "2020-05-19 14:33:41,883 Epoch  13 Step:    25800 Batch Loss:     1.843168 Tokens per Sec:     4582, Lr: 0.000300\n",
            "2020-05-19 14:34:30,606 Epoch  13 Step:    25900 Batch Loss:     1.162993 Tokens per Sec:     4765, Lr: 0.000300\n",
            "2020-05-19 14:35:18,889 Epoch  13 Step:    26000 Batch Loss:     0.981001 Tokens per Sec:     4640, Lr: 0.000300\n",
            "2020-05-19 14:36:21,891 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:36:21,892 Saving new checkpoint.\n",
            "2020-05-19 14:36:22,335 Example #0\n",
            "2020-05-19 14:36:22,335 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:36:22,335 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:36:22,335 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:36:22,335 Example #1\n",
            "2020-05-19 14:36:22,336 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:36:22,336 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:36:22,336 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:36:22,336 Example #2\n",
            "2020-05-19 14:36:22,336 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:36:22,336 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:36:22,336 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:36:22,336 Example #3\n",
            "2020-05-19 14:36:22,337 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:36:22,337 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:36:22,337 \tHypothesis: “ Ni ḓo guma lini u ṱudza kha mihumbulo mivhili yo fhamba - fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:36:22,337 Validation result (greedy) at epoch  13, step    26000: bleu:  37.52, loss: 31805.8945, ppl:   3.9118, duration: 63.4471s\n",
            "2020-05-19 14:37:10,454 Epoch  13 Step:    26100 Batch Loss:     1.213585 Tokens per Sec:     4699, Lr: 0.000300\n",
            "2020-05-19 14:37:58,788 Epoch  13 Step:    26200 Batch Loss:     1.526292 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 14:38:47,060 Epoch  13 Step:    26300 Batch Loss:     1.393700 Tokens per Sec:     4627, Lr: 0.000300\n",
            "2020-05-19 14:39:35,367 Epoch  13 Step:    26400 Batch Loss:     1.432562 Tokens per Sec:     4590, Lr: 0.000300\n",
            "2020-05-19 14:40:24,185 Epoch  13 Step:    26500 Batch Loss:     1.516371 Tokens per Sec:     4668, Lr: 0.000300\n",
            "2020-05-19 14:41:12,519 Epoch  13 Step:    26600 Batch Loss:     1.299774 Tokens per Sec:     4668, Lr: 0.000300\n",
            "2020-05-19 14:42:01,410 Epoch  13 Step:    26700 Batch Loss:     1.473685 Tokens per Sec:     4668, Lr: 0.000300\n",
            "2020-05-19 14:42:49,735 Epoch  13 Step:    26800 Batch Loss:     1.397445 Tokens per Sec:     4630, Lr: 0.000300\n",
            "2020-05-19 14:43:37,751 Epoch  13 Step:    26900 Batch Loss:     1.144962 Tokens per Sec:     4571, Lr: 0.000300\n",
            "2020-05-19 14:44:25,671 Epoch  13 Step:    27000 Batch Loss:     1.764435 Tokens per Sec:     4512, Lr: 0.000300\n",
            "2020-05-19 14:45:20,976 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:45:20,976 Saving new checkpoint.\n",
            "2020-05-19 14:45:21,479 Example #0\n",
            "2020-05-19 14:45:21,479 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:45:21,479 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:45:21,480 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:45:21,480 Example #1\n",
            "2020-05-19 14:45:21,480 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:45:21,480 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:45:21,480 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:45:21,480 Example #2\n",
            "2020-05-19 14:45:21,480 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:45:21,481 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:45:21,481 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:45:21,481 Example #3\n",
            "2020-05-19 14:45:21,481 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:45:21,481 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:45:21,481 \tHypothesis: “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe mihumbulo mivhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:45:21,481 Validation result (greedy) at epoch  13, step    27000: bleu:  37.93, loss: 31465.8516, ppl:   3.8552, duration: 55.8095s\n",
            "2020-05-19 14:46:10,445 Epoch  13 Step:    27100 Batch Loss:     1.389953 Tokens per Sec:     4675, Lr: 0.000300\n",
            "2020-05-19 14:46:59,272 Epoch  13 Step:    27200 Batch Loss:     1.330255 Tokens per Sec:     4650, Lr: 0.000300\n",
            "2020-05-19 14:47:47,314 Epoch  13 Step:    27300 Batch Loss:     1.097317 Tokens per Sec:     4567, Lr: 0.000300\n",
            "2020-05-19 14:48:13,508 Epoch  13: total training loss 2893.93\n",
            "2020-05-19 14:48:13,509 EPOCH 14\n",
            "2020-05-19 14:48:36,013 Epoch  14 Step:    27400 Batch Loss:     1.120734 Tokens per Sec:     4604, Lr: 0.000300\n",
            "2020-05-19 14:49:24,343 Epoch  14 Step:    27500 Batch Loss:     1.547631 Tokens per Sec:     4648, Lr: 0.000300\n",
            "2020-05-19 14:50:12,911 Epoch  14 Step:    27600 Batch Loss:     1.297401 Tokens per Sec:     4712, Lr: 0.000300\n",
            "2020-05-19 14:51:01,952 Epoch  14 Step:    27700 Batch Loss:     1.478125 Tokens per Sec:     4643, Lr: 0.000300\n",
            "2020-05-19 14:51:50,662 Epoch  14 Step:    27800 Batch Loss:     1.294140 Tokens per Sec:     4578, Lr: 0.000300\n",
            "2020-05-19 14:52:38,819 Epoch  14 Step:    27900 Batch Loss:     1.318087 Tokens per Sec:     4636, Lr: 0.000300\n",
            "2020-05-19 14:53:27,693 Epoch  14 Step:    28000 Batch Loss:     1.459895 Tokens per Sec:     4596, Lr: 0.000300\n",
            "2020-05-19 14:54:23,330 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 14:54:23,330 Saving new checkpoint.\n",
            "2020-05-19 14:54:23,804 Example #0\n",
            "2020-05-19 14:54:23,804 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 14:54:23,804 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 14:54:23,805 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 14:54:23,805 Example #1\n",
            "2020-05-19 14:54:23,805 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 14:54:23,805 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 14:54:23,805 \tHypothesis: O konḓelela maṱungu manzhi , fhedzi ho ngo vhuya a kundelwa u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 14:54:23,805 Example #2\n",
            "2020-05-19 14:54:23,806 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 14:54:23,806 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 14:54:23,806 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 14:54:23,806 Example #3\n",
            "2020-05-19 14:54:23,806 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 14:54:23,806 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 14:54:23,807 \tHypothesis: “ Ni ḓo ṱudza lini hune na dzula hone kha mahumbulele mavhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 14:54:23,807 Validation result (greedy) at epoch  14, step    28000: bleu:  37.91, loss: 31105.4355, ppl:   3.7961, duration: 56.1133s\n",
            "2020-05-19 14:55:11,865 Epoch  14 Step:    28100 Batch Loss:     1.578619 Tokens per Sec:     4715, Lr: 0.000300\n",
            "2020-05-19 14:56:00,247 Epoch  14 Step:    28200 Batch Loss:     1.407461 Tokens per Sec:     4684, Lr: 0.000300\n",
            "2020-05-19 14:56:48,523 Epoch  14 Step:    28300 Batch Loss:     1.515860 Tokens per Sec:     4699, Lr: 0.000300\n",
            "2020-05-19 14:57:36,787 Epoch  14 Step:    28400 Batch Loss:     1.176191 Tokens per Sec:     4607, Lr: 0.000300\n",
            "2020-05-19 14:58:25,249 Epoch  14 Step:    28500 Batch Loss:     1.365945 Tokens per Sec:     4678, Lr: 0.000300\n",
            "2020-05-19 14:59:13,561 Epoch  14 Step:    28600 Batch Loss:     1.435717 Tokens per Sec:     4526, Lr: 0.000300\n",
            "2020-05-19 15:00:01,318 Epoch  14 Step:    28700 Batch Loss:     1.486105 Tokens per Sec:     4676, Lr: 0.000300\n",
            "2020-05-19 15:00:49,733 Epoch  14 Step:    28800 Batch Loss:     1.156011 Tokens per Sec:     4583, Lr: 0.000300\n",
            "2020-05-19 15:01:37,759 Epoch  14 Step:    28900 Batch Loss:     1.190224 Tokens per Sec:     4574, Lr: 0.000300\n",
            "2020-05-19 15:02:25,605 Epoch  14 Step:    29000 Batch Loss:     1.216319 Tokens per Sec:     4693, Lr: 0.000300\n",
            "2020-05-19 15:03:17,415 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:03:17,415 Saving new checkpoint.\n",
            "2020-05-19 15:03:17,920 Example #0\n",
            "2020-05-19 15:03:17,920 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:03:17,920 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:03:17,920 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:03:17,921 Example #1\n",
            "2020-05-19 15:03:17,921 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:03:17,921 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:03:17,921 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa athu vhuya a fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:03:17,921 Example #2\n",
            "2020-05-19 15:03:17,921 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:03:17,921 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:03:17,922 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo .\n",
            "2020-05-19 15:03:17,922 Example #3\n",
            "2020-05-19 15:03:17,922 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:03:17,922 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:03:17,922 \tHypothesis: “ Ni ḓo fhedza tshifhinga tshingafhani ni tshi khou ṱudza thungo dzoṱhe mihumbulo yo fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:03:17,922 Validation result (greedy) at epoch  14, step    29000: bleu:  38.02, loss: 30879.6074, ppl:   3.7595, duration: 52.3169s\n",
            "2020-05-19 15:04:06,165 Epoch  14 Step:    29100 Batch Loss:     1.462249 Tokens per Sec:     4571, Lr: 0.000300\n",
            "2020-05-19 15:04:54,091 Epoch  14 Step:    29200 Batch Loss:     1.441350 Tokens per Sec:     4588, Lr: 0.000300\n",
            "2020-05-19 15:05:42,452 Epoch  14 Step:    29300 Batch Loss:     1.665146 Tokens per Sec:     4708, Lr: 0.000300\n",
            "2020-05-19 15:06:30,129 Epoch  14 Step:    29400 Batch Loss:     1.171231 Tokens per Sec:     4550, Lr: 0.000300\n",
            "2020-05-19 15:06:58,221 Epoch  14: total training loss 2847.98\n",
            "2020-05-19 15:06:58,221 EPOCH 15\n",
            "2020-05-19 15:07:18,909 Epoch  15 Step:    29500 Batch Loss:     1.095823 Tokens per Sec:     4569, Lr: 0.000300\n",
            "2020-05-19 15:08:07,630 Epoch  15 Step:    29600 Batch Loss:     1.245470 Tokens per Sec:     4636, Lr: 0.000300\n",
            "2020-05-19 15:08:55,727 Epoch  15 Step:    29700 Batch Loss:     1.271278 Tokens per Sec:     4653, Lr: 0.000300\n",
            "2020-05-19 15:09:44,727 Epoch  15 Step:    29800 Batch Loss:     1.441488 Tokens per Sec:     4713, Lr: 0.000300\n",
            "2020-05-19 15:10:33,308 Epoch  15 Step:    29900 Batch Loss:     1.283909 Tokens per Sec:     4663, Lr: 0.000300\n",
            "2020-05-19 15:11:21,212 Epoch  15 Step:    30000 Batch Loss:     1.162260 Tokens per Sec:     4655, Lr: 0.000300\n",
            "2020-05-19 15:12:11,449 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:12:11,450 Saving new checkpoint.\n",
            "2020-05-19 15:12:11,971 Example #0\n",
            "2020-05-19 15:12:11,971 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:12:11,971 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:12:11,972 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:12:11,972 Example #1\n",
            "2020-05-19 15:12:11,972 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:12:11,972 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:12:11,972 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:12:11,972 Example #2\n",
            "2020-05-19 15:12:11,972 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:12:11,972 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:12:11,973 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo .\n",
            "2020-05-19 15:12:11,973 Example #3\n",
            "2020-05-19 15:12:11,973 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:12:11,973 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:12:11,973 \tHypothesis: “ Ni ḓo litsha hani u ṱudza kha mavhonele mavhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:12:11,973 Validation result (greedy) at epoch  15, step    30000: bleu:  38.72, loss: 30723.8691, ppl:   3.7345, duration: 50.7607s\n",
            "2020-05-19 15:13:00,069 Epoch  15 Step:    30100 Batch Loss:     1.518086 Tokens per Sec:     4669, Lr: 0.000300\n",
            "2020-05-19 15:13:48,729 Epoch  15 Step:    30200 Batch Loss:     1.423849 Tokens per Sec:     4692, Lr: 0.000300\n",
            "2020-05-19 15:14:37,137 Epoch  15 Step:    30300 Batch Loss:     1.346053 Tokens per Sec:     4541, Lr: 0.000300\n",
            "2020-05-19 15:15:25,829 Epoch  15 Step:    30400 Batch Loss:     1.382568 Tokens per Sec:     4710, Lr: 0.000300\n",
            "2020-05-19 15:16:14,492 Epoch  15 Step:    30500 Batch Loss:     1.169802 Tokens per Sec:     4585, Lr: 0.000300\n",
            "2020-05-19 15:17:02,149 Epoch  15 Step:    30600 Batch Loss:     1.142358 Tokens per Sec:     4541, Lr: 0.000300\n",
            "2020-05-19 15:17:50,736 Epoch  15 Step:    30700 Batch Loss:     1.081159 Tokens per Sec:     4689, Lr: 0.000300\n",
            "2020-05-19 15:18:38,470 Epoch  15 Step:    30800 Batch Loss:     1.418908 Tokens per Sec:     4558, Lr: 0.000300\n",
            "2020-05-19 15:19:26,240 Epoch  15 Step:    30900 Batch Loss:     1.376020 Tokens per Sec:     4610, Lr: 0.000300\n",
            "2020-05-19 15:20:14,280 Epoch  15 Step:    31000 Batch Loss:     1.196414 Tokens per Sec:     4708, Lr: 0.000300\n",
            "2020-05-19 15:21:07,108 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:21:07,109 Saving new checkpoint.\n",
            "2020-05-19 15:21:07,701 Example #0\n",
            "2020-05-19 15:21:07,701 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:21:07,701 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:21:07,701 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:21:07,702 Example #1\n",
            "2020-05-19 15:21:07,702 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:21:07,702 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:21:07,702 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa tsha fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 15:21:07,702 Example #2\n",
            "2020-05-19 15:21:07,703 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:21:07,703 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:21:07,703 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 15:21:07,703 Example #3\n",
            "2020-05-19 15:21:07,703 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:21:07,703 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:21:07,704 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe kha mihumbulo mivhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:21:07,704 Validation result (greedy) at epoch  15, step    31000: bleu:  38.48, loss: 30693.8535, ppl:   3.7297, duration: 53.4236s\n",
            "2020-05-19 15:21:56,351 Epoch  15 Step:    31100 Batch Loss:     1.248412 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 15:22:44,903 Epoch  15 Step:    31200 Batch Loss:     1.140726 Tokens per Sec:     4631, Lr: 0.000300\n",
            "2020-05-19 15:23:32,605 Epoch  15 Step:    31300 Batch Loss:     1.355419 Tokens per Sec:     4531, Lr: 0.000300\n",
            "2020-05-19 15:24:20,838 Epoch  15 Step:    31400 Batch Loss:     1.177468 Tokens per Sec:     4642, Lr: 0.000300\n",
            "2020-05-19 15:25:09,029 Epoch  15 Step:    31500 Batch Loss:     1.273577 Tokens per Sec:     4720, Lr: 0.000300\n",
            "2020-05-19 15:25:39,119 Epoch  15: total training loss 2794.85\n",
            "2020-05-19 15:25:39,119 EPOCH 16\n",
            "2020-05-19 15:25:58,028 Epoch  16 Step:    31600 Batch Loss:     1.362220 Tokens per Sec:     4511, Lr: 0.000300\n",
            "2020-05-19 15:26:46,318 Epoch  16 Step:    31700 Batch Loss:     1.626305 Tokens per Sec:     4627, Lr: 0.000300\n",
            "2020-05-19 15:27:34,710 Epoch  16 Step:    31800 Batch Loss:     1.248674 Tokens per Sec:     4564, Lr: 0.000300\n",
            "2020-05-19 15:28:23,524 Epoch  16 Step:    31900 Batch Loss:     1.535686 Tokens per Sec:     4610, Lr: 0.000300\n",
            "2020-05-19 15:29:12,198 Epoch  16 Step:    32000 Batch Loss:     1.478457 Tokens per Sec:     4571, Lr: 0.000300\n",
            "2020-05-19 15:30:00,122 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:30:00,122 Saving new checkpoint.\n",
            "2020-05-19 15:30:00,679 Example #0\n",
            "2020-05-19 15:30:00,680 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:30:00,680 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:30:00,680 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:30:00,680 Example #1\n",
            "2020-05-19 15:30:00,680 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:30:00,680 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:30:00,680 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 15:30:00,680 Example #2\n",
            "2020-05-19 15:30:00,681 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:30:00,681 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:30:00,681 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 15:30:00,681 Example #3\n",
            "2020-05-19 15:30:00,681 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:30:00,682 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:30:00,682 \tHypothesis: “ Ni ḓo litsha lini u ṱudza kha mihumbulo mivhili yo fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:30:00,682 Validation result (greedy) at epoch  16, step    32000: bleu:  38.91, loss: 30646.2305, ppl:   3.7220, duration: 48.4834s\n",
            "2020-05-19 15:30:49,669 Epoch  16 Step:    32100 Batch Loss:     1.360562 Tokens per Sec:     4547, Lr: 0.000300\n",
            "2020-05-19 15:31:37,867 Epoch  16 Step:    32200 Batch Loss:     1.213731 Tokens per Sec:     4549, Lr: 0.000300\n",
            "2020-05-19 15:32:26,465 Epoch  16 Step:    32300 Batch Loss:     1.247423 Tokens per Sec:     4622, Lr: 0.000300\n",
            "2020-05-19 15:33:15,586 Epoch  16 Step:    32400 Batch Loss:     1.645640 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 15:34:04,381 Epoch  16 Step:    32500 Batch Loss:     1.211247 Tokens per Sec:     4576, Lr: 0.000300\n",
            "2020-05-19 15:34:53,490 Epoch  16 Step:    32600 Batch Loss:     1.400913 Tokens per Sec:     4581, Lr: 0.000300\n",
            "2020-05-19 15:35:42,473 Epoch  16 Step:    32700 Batch Loss:     1.286158 Tokens per Sec:     4707, Lr: 0.000300\n",
            "2020-05-19 15:36:30,770 Epoch  16 Step:    32800 Batch Loss:     1.511631 Tokens per Sec:     4495, Lr: 0.000300\n",
            "2020-05-19 15:37:19,856 Epoch  16 Step:    32900 Batch Loss:     1.364190 Tokens per Sec:     4693, Lr: 0.000300\n",
            "2020-05-19 15:38:08,439 Epoch  16 Step:    33000 Batch Loss:     1.476517 Tokens per Sec:     4534, Lr: 0.000300\n",
            "2020-05-19 15:38:56,981 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:38:56,981 Saving new checkpoint.\n",
            "2020-05-19 15:38:57,435 Example #0\n",
            "2020-05-19 15:38:57,435 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:38:57,435 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:38:57,436 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:38:57,436 Example #1\n",
            "2020-05-19 15:38:57,436 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:38:57,436 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:38:57,436 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa tsha fhisea kana u vha hawe na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:38:57,436 Example #2\n",
            "2020-05-19 15:38:57,437 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:38:57,437 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:38:57,437 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 15:38:57,437 Example #3\n",
            "2020-05-19 15:38:57,437 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:38:57,437 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:38:57,437 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe dza mavhonele mavhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:38:57,438 Validation result (greedy) at epoch  16, step    33000: bleu:  38.64, loss: 30271.5547, ppl:   3.6627, duration: 48.9978s\n",
            "2020-05-19 15:39:46,214 Epoch  16 Step:    33100 Batch Loss:     0.982009 Tokens per Sec:     4638, Lr: 0.000300\n",
            "2020-05-19 15:40:34,684 Epoch  16 Step:    33200 Batch Loss:     1.369248 Tokens per Sec:     4532, Lr: 0.000300\n",
            "2020-05-19 15:41:23,606 Epoch  16 Step:    33300 Batch Loss:     1.133521 Tokens per Sec:     4689, Lr: 0.000300\n",
            "2020-05-19 15:42:12,118 Epoch  16 Step:    33400 Batch Loss:     1.455001 Tokens per Sec:     4530, Lr: 0.000300\n",
            "2020-05-19 15:43:00,916 Epoch  16 Step:    33500 Batch Loss:     1.553172 Tokens per Sec:     4628, Lr: 0.000300\n",
            "2020-05-19 15:43:49,623 Epoch  16 Step:    33600 Batch Loss:     1.137857 Tokens per Sec:     4624, Lr: 0.000300\n",
            "2020-05-19 15:44:21,684 Epoch  16: total training loss 2737.96\n",
            "2020-05-19 15:44:21,684 EPOCH 17\n",
            "2020-05-19 15:44:38,056 Epoch  17 Step:    33700 Batch Loss:     1.337060 Tokens per Sec:     4461, Lr: 0.000300\n",
            "2020-05-19 15:45:26,251 Epoch  17 Step:    33800 Batch Loss:     1.435604 Tokens per Sec:     4630, Lr: 0.000300\n",
            "2020-05-19 15:46:14,806 Epoch  17 Step:    33900 Batch Loss:     1.240700 Tokens per Sec:     4801, Lr: 0.000300\n",
            "2020-05-19 15:47:03,064 Epoch  17 Step:    34000 Batch Loss:     1.293840 Tokens per Sec:     4566, Lr: 0.000300\n",
            "2020-05-19 15:48:09,707 Example #0\n",
            "2020-05-19 15:48:09,708 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:48:09,708 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:48:09,708 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:48:09,708 Example #1\n",
            "2020-05-19 15:48:09,708 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:48:09,708 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:48:09,708 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa tsha fhisetshela u fhisea hawe kana u vha hawe na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:48:09,709 Example #2\n",
            "2020-05-19 15:48:09,709 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:48:09,709 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:48:09,709 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 15:48:09,709 Example #3\n",
            "2020-05-19 15:48:09,709 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:48:09,709 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:48:09,710 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe mihumbulo yo fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:48:09,710 Validation result (greedy) at epoch  17, step    34000: bleu:  38.35, loss: 30388.9453, ppl:   3.6812, duration: 66.6455s\n",
            "2020-05-19 15:48:58,032 Epoch  17 Step:    34100 Batch Loss:     1.307009 Tokens per Sec:     4657, Lr: 0.000300\n",
            "2020-05-19 15:49:46,262 Epoch  17 Step:    34200 Batch Loss:     1.326108 Tokens per Sec:     4666, Lr: 0.000300\n",
            "2020-05-19 15:50:34,886 Epoch  17 Step:    34300 Batch Loss:     1.363312 Tokens per Sec:     4714, Lr: 0.000300\n",
            "2020-05-19 15:51:23,265 Epoch  17 Step:    34400 Batch Loss:     1.519357 Tokens per Sec:     4641, Lr: 0.000300\n",
            "2020-05-19 15:52:11,548 Epoch  17 Step:    34500 Batch Loss:     1.554922 Tokens per Sec:     4664, Lr: 0.000300\n",
            "2020-05-19 15:52:59,679 Epoch  17 Step:    34600 Batch Loss:     1.303999 Tokens per Sec:     4682, Lr: 0.000300\n",
            "2020-05-19 15:53:47,749 Epoch  17 Step:    34700 Batch Loss:     1.059292 Tokens per Sec:     4634, Lr: 0.000300\n",
            "2020-05-19 15:54:36,347 Epoch  17 Step:    34800 Batch Loss:     1.505344 Tokens per Sec:     4656, Lr: 0.000300\n",
            "2020-05-19 15:55:23,992 Epoch  17 Step:    34900 Batch Loss:     1.178607 Tokens per Sec:     4623, Lr: 0.000300\n",
            "2020-05-19 15:56:12,751 Epoch  17 Step:    35000 Batch Loss:     0.984365 Tokens per Sec:     4569, Lr: 0.000300\n",
            "2020-05-19 15:57:01,543 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 15:57:01,543 Saving new checkpoint.\n",
            "2020-05-19 15:57:01,992 Example #0\n",
            "2020-05-19 15:57:01,992 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 15:57:01,992 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 15:57:01,992 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 15:57:01,992 Example #1\n",
            "2020-05-19 15:57:01,993 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 15:57:01,993 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:57:01,993 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa tsha fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 15:57:01,993 Example #2\n",
            "2020-05-19 15:57:01,993 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 15:57:01,993 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 15:57:01,994 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 15:57:01,994 Example #3\n",
            "2020-05-19 15:57:01,994 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 15:57:01,994 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 15:57:01,994 \tHypothesis: “ Ni ḓo guma lini u ṱudza kha mihumbulo mivhili yo fhamba - fhambanaho ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 15:57:01,994 Validation result (greedy) at epoch  17, step    35000: bleu:  39.18, loss: 30051.5859, ppl:   3.6283, duration: 49.2424s\n",
            "2020-05-19 15:57:50,251 Epoch  17 Step:    35100 Batch Loss:     1.326593 Tokens per Sec:     4704, Lr: 0.000300\n",
            "2020-05-19 15:58:38,486 Epoch  17 Step:    35200 Batch Loss:     1.351532 Tokens per Sec:     4629, Lr: 0.000300\n",
            "2020-05-19 15:59:26,376 Epoch  17 Step:    35300 Batch Loss:     0.998021 Tokens per Sec:     4631, Lr: 0.000300\n",
            "2020-05-19 16:00:15,075 Epoch  17 Step:    35400 Batch Loss:     1.228718 Tokens per Sec:     4622, Lr: 0.000300\n",
            "2020-05-19 16:01:03,572 Epoch  17 Step:    35500 Batch Loss:     1.225340 Tokens per Sec:     4580, Lr: 0.000300\n",
            "2020-05-19 16:01:51,416 Epoch  17 Step:    35600 Batch Loss:     1.271234 Tokens per Sec:     4582, Lr: 0.000300\n",
            "2020-05-19 16:02:39,365 Epoch  17 Step:    35700 Batch Loss:     1.179405 Tokens per Sec:     4548, Lr: 0.000300\n",
            "2020-05-19 16:03:14,498 Epoch  17: total training loss 2703.86\n",
            "2020-05-19 16:03:14,499 EPOCH 18\n",
            "2020-05-19 16:03:28,095 Epoch  18 Step:    35800 Batch Loss:     1.060081 Tokens per Sec:     4656, Lr: 0.000300\n",
            "2020-05-19 16:04:16,662 Epoch  18 Step:    35900 Batch Loss:     1.086687 Tokens per Sec:     4612, Lr: 0.000300\n",
            "2020-05-19 16:05:05,479 Epoch  18 Step:    36000 Batch Loss:     1.318810 Tokens per Sec:     4667, Lr: 0.000300\n",
            "2020-05-19 16:05:57,049 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:05:57,049 Saving new checkpoint.\n",
            "2020-05-19 16:05:57,488 Example #0\n",
            "2020-05-19 16:05:57,488 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:05:57,488 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:05:57,488 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:05:57,488 Example #1\n",
            "2020-05-19 16:05:57,489 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:05:57,489 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:05:57,489 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:05:57,489 Example #2\n",
            "2020-05-19 16:05:57,489 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:05:57,489 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:05:57,489 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:05:57,489 Example #3\n",
            "2020-05-19 16:05:57,490 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:05:57,490 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:05:57,490 \tHypothesis: “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe . ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 16:05:57,490 Validation result (greedy) at epoch  18, step    36000: bleu:  39.35, loss: 29892.0625, ppl:   3.6036, duration: 52.0105s\n",
            "2020-05-19 16:06:45,898 Epoch  18 Step:    36100 Batch Loss:     1.423036 Tokens per Sec:     4624, Lr: 0.000300\n",
            "2020-05-19 16:07:34,423 Epoch  18 Step:    36200 Batch Loss:     1.196442 Tokens per Sec:     4681, Lr: 0.000300\n",
            "2020-05-19 16:08:22,505 Epoch  18 Step:    36300 Batch Loss:     1.110568 Tokens per Sec:     4447, Lr: 0.000300\n",
            "2020-05-19 16:09:10,981 Epoch  18 Step:    36400 Batch Loss:     0.978227 Tokens per Sec:     4624, Lr: 0.000300\n",
            "2020-05-19 16:09:58,423 Epoch  18 Step:    36500 Batch Loss:     1.097114 Tokens per Sec:     4534, Lr: 0.000300\n",
            "2020-05-19 16:10:46,902 Epoch  18 Step:    36600 Batch Loss:     1.294136 Tokens per Sec:     4592, Lr: 0.000300\n",
            "2020-05-19 16:11:35,522 Epoch  18 Step:    36700 Batch Loss:     1.203169 Tokens per Sec:     4700, Lr: 0.000300\n",
            "2020-05-19 16:12:23,836 Epoch  18 Step:    36800 Batch Loss:     1.444497 Tokens per Sec:     4617, Lr: 0.000300\n",
            "2020-05-19 16:13:12,353 Epoch  18 Step:    36900 Batch Loss:     1.359952 Tokens per Sec:     4578, Lr: 0.000300\n",
            "2020-05-19 16:14:01,103 Epoch  18 Step:    37000 Batch Loss:     1.220583 Tokens per Sec:     4713, Lr: 0.000300\n",
            "2020-05-19 16:14:51,783 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:14:51,783 Saving new checkpoint.\n",
            "2020-05-19 16:14:52,213 Example #0\n",
            "2020-05-19 16:14:52,213 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:14:52,213 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:14:52,213 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:14:52,213 Example #1\n",
            "2020-05-19 16:14:52,214 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:14:52,214 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:14:52,214 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo tsha fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 16:14:52,214 Example #2\n",
            "2020-05-19 16:14:52,214 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:14:52,214 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:14:52,214 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:14:52,214 Example #3\n",
            "2020-05-19 16:14:52,215 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:14:52,215 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:14:52,215 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe dza mihumbulo mivhili ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 16:14:52,215 Validation result (greedy) at epoch  18, step    37000: bleu:  39.45, loss: 29705.0449, ppl:   3.5748, duration: 51.1117s\n",
            "2020-05-19 16:15:40,519 Epoch  18 Step:    37100 Batch Loss:     1.190973 Tokens per Sec:     4696, Lr: 0.000300\n",
            "2020-05-19 16:16:28,535 Epoch  18 Step:    37200 Batch Loss:     1.200631 Tokens per Sec:     4652, Lr: 0.000300\n",
            "2020-05-19 16:17:17,036 Epoch  18 Step:    37300 Batch Loss:     1.214116 Tokens per Sec:     4587, Lr: 0.000300\n",
            "2020-05-19 16:18:05,659 Epoch  18 Step:    37400 Batch Loss:     0.998693 Tokens per Sec:     4725, Lr: 0.000300\n",
            "2020-05-19 16:18:53,312 Epoch  18 Step:    37500 Batch Loss:     1.550203 Tokens per Sec:     4698, Lr: 0.000300\n",
            "2020-05-19 16:19:41,445 Epoch  18 Step:    37600 Batch Loss:     1.410833 Tokens per Sec:     4683, Lr: 0.000300\n",
            "2020-05-19 16:20:29,904 Epoch  18 Step:    37700 Batch Loss:     1.409302 Tokens per Sec:     4565, Lr: 0.000300\n",
            "2020-05-19 16:21:18,614 Epoch  18 Step:    37800 Batch Loss:     1.369534 Tokens per Sec:     4660, Lr: 0.000300\n",
            "2020-05-19 16:21:54,248 Epoch  18: total training loss 2649.24\n",
            "2020-05-19 16:21:54,248 EPOCH 19\n",
            "2020-05-19 16:22:07,842 Epoch  19 Step:    37900 Batch Loss:     1.058246 Tokens per Sec:     4593, Lr: 0.000300\n",
            "2020-05-19 16:22:56,502 Epoch  19 Step:    38000 Batch Loss:     1.353798 Tokens per Sec:     4574, Lr: 0.000300\n",
            "2020-05-19 16:23:47,347 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:23:47,348 Saving new checkpoint.\n",
            "2020-05-19 16:23:47,866 Example #0\n",
            "2020-05-19 16:23:47,866 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:23:47,866 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:23:47,867 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:23:47,867 Example #1\n",
            "2020-05-19 16:23:47,867 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:23:47,867 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:23:47,867 \tHypothesis: O konḓelela vhuleme vhunzhi , naho o vha a sa athu vhuya a litsha u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:23:47,867 Example #2\n",
            "2020-05-19 16:23:47,867 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:23:47,868 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:23:47,868 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:23:47,868 Example #3\n",
            "2020-05-19 16:23:47,868 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:23:47,868 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:23:47,868 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 16:23:47,868 Validation result (greedy) at epoch  19, step    38000: bleu:  39.82, loss: 29640.8770, ppl:   3.5650, duration: 51.3657s\n",
            "2020-05-19 16:24:36,510 Epoch  19 Step:    38100 Batch Loss:     1.192388 Tokens per Sec:     4619, Lr: 0.000300\n",
            "2020-05-19 16:25:24,780 Epoch  19 Step:    38200 Batch Loss:     1.051443 Tokens per Sec:     4605, Lr: 0.000300\n",
            "2020-05-19 16:26:12,970 Epoch  19 Step:    38300 Batch Loss:     1.188721 Tokens per Sec:     4634, Lr: 0.000300\n",
            "2020-05-19 16:27:02,010 Epoch  19 Step:    38400 Batch Loss:     1.355304 Tokens per Sec:     4627, Lr: 0.000300\n",
            "2020-05-19 16:27:49,987 Epoch  19 Step:    38500 Batch Loss:     1.127310 Tokens per Sec:     4647, Lr: 0.000300\n",
            "2020-05-19 16:28:38,793 Epoch  19 Step:    38600 Batch Loss:     1.070264 Tokens per Sec:     4717, Lr: 0.000300\n",
            "2020-05-19 16:29:27,909 Epoch  19 Step:    38700 Batch Loss:     1.375622 Tokens per Sec:     4765, Lr: 0.000300\n",
            "2020-05-19 16:30:15,988 Epoch  19 Step:    38800 Batch Loss:     1.113519 Tokens per Sec:     4602, Lr: 0.000300\n",
            "2020-05-19 16:31:04,362 Epoch  19 Step:    38900 Batch Loss:     1.272863 Tokens per Sec:     4609, Lr: 0.000300\n",
            "2020-05-19 16:31:53,062 Epoch  19 Step:    39000 Batch Loss:     0.995208 Tokens per Sec:     4733, Lr: 0.000300\n",
            "2020-05-19 16:32:51,455 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:32:51,456 Saving new checkpoint.\n",
            "2020-05-19 16:32:51,911 Example #0\n",
            "2020-05-19 16:32:51,912 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:32:51,912 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:32:51,912 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:32:51,912 Example #1\n",
            "2020-05-19 16:32:51,913 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:32:51,913 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:32:51,913 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:32:51,913 Example #2\n",
            "2020-05-19 16:32:51,913 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:32:51,913 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:32:51,913 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:32:51,913 Example #3\n",
            "2020-05-19 16:32:51,914 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:32:51,914 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:32:51,914 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe ? ” — 1 VHA - saina saina 18 : 21 .\n",
            "2020-05-19 16:32:51,914 Validation result (greedy) at epoch  19, step    39000: bleu:  39.54, loss: 29443.3496, ppl:   3.5349, duration: 58.8514s\n",
            "2020-05-19 16:33:40,545 Epoch  19 Step:    39100 Batch Loss:     1.169011 Tokens per Sec:     4625, Lr: 0.000300\n",
            "2020-05-19 16:34:28,323 Epoch  19 Step:    39200 Batch Loss:     1.004086 Tokens per Sec:     4572, Lr: 0.000300\n",
            "2020-05-19 16:35:16,827 Epoch  19 Step:    39300 Batch Loss:     1.201625 Tokens per Sec:     4558, Lr: 0.000300\n",
            "2020-05-19 16:36:05,655 Epoch  19 Step:    39400 Batch Loss:     1.055228 Tokens per Sec:     4620, Lr: 0.000300\n",
            "2020-05-19 16:36:53,984 Epoch  19 Step:    39500 Batch Loss:     1.042062 Tokens per Sec:     4556, Lr: 0.000300\n",
            "2020-05-19 16:37:42,966 Epoch  19 Step:    39600 Batch Loss:     0.971376 Tokens per Sec:     4626, Lr: 0.000300\n",
            "2020-05-19 16:38:31,769 Epoch  19 Step:    39700 Batch Loss:     1.445066 Tokens per Sec:     4639, Lr: 0.000300\n",
            "2020-05-19 16:39:20,094 Epoch  19 Step:    39800 Batch Loss:     1.147318 Tokens per Sec:     4626, Lr: 0.000300\n",
            "2020-05-19 16:40:08,724 Epoch  19 Step:    39900 Batch Loss:     1.361335 Tokens per Sec:     4611, Lr: 0.000300\n",
            "2020-05-19 16:40:42,692 Epoch  19: total training loss 2606.81\n",
            "2020-05-19 16:40:42,693 EPOCH 20\n",
            "2020-05-19 16:40:57,463 Epoch  20 Step:    40000 Batch Loss:     1.101279 Tokens per Sec:     4548, Lr: 0.000300\n",
            "2020-05-19 16:41:46,756 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:41:46,756 Saving new checkpoint.\n",
            "2020-05-19 16:41:47,237 Example #0\n",
            "2020-05-19 16:41:47,237 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:41:47,237 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:41:47,237 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:41:47,237 Example #1\n",
            "2020-05-19 16:41:47,238 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:41:47,238 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:41:47,238 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea hawe kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:41:47,238 Example #2\n",
            "2020-05-19 16:41:47,238 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:41:47,238 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:41:47,238 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:41:47,238 Example #3\n",
            "2020-05-19 16:41:47,239 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:41:47,239 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:41:47,239 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe dza mihumbulo mivhili ? ” — 1 VHA - HOSI 18 : 21 .\n",
            "2020-05-19 16:41:47,239 Validation result (greedy) at epoch  20, step    40000: bleu:  39.89, loss: 29347.5742, ppl:   3.5204, duration: 49.7754s\n",
            "2020-05-19 16:42:36,163 Epoch  20 Step:    40100 Batch Loss:     1.186174 Tokens per Sec:     4637, Lr: 0.000300\n",
            "2020-05-19 16:43:24,481 Epoch  20 Step:    40200 Batch Loss:     1.394699 Tokens per Sec:     4515, Lr: 0.000300\n",
            "2020-05-19 16:44:13,330 Epoch  20 Step:    40300 Batch Loss:     1.192940 Tokens per Sec:     4505, Lr: 0.000300\n",
            "2020-05-19 16:45:01,575 Epoch  20 Step:    40400 Batch Loss:     1.155409 Tokens per Sec:     4528, Lr: 0.000300\n",
            "2020-05-19 16:45:50,281 Epoch  20 Step:    40500 Batch Loss:     1.302750 Tokens per Sec:     4707, Lr: 0.000300\n",
            "2020-05-19 16:46:39,220 Epoch  20 Step:    40600 Batch Loss:     1.273804 Tokens per Sec:     4689, Lr: 0.000300\n",
            "2020-05-19 16:47:28,061 Epoch  20 Step:    40700 Batch Loss:     1.300528 Tokens per Sec:     4607, Lr: 0.000300\n",
            "2020-05-19 16:48:16,371 Epoch  20 Step:    40800 Batch Loss:     1.179304 Tokens per Sec:     4554, Lr: 0.000300\n",
            "2020-05-19 16:49:05,184 Epoch  20 Step:    40900 Batch Loss:     1.216538 Tokens per Sec:     4700, Lr: 0.000300\n",
            "2020-05-19 16:49:53,843 Epoch  20 Step:    41000 Batch Loss:     1.165207 Tokens per Sec:     4600, Lr: 0.000300\n",
            "2020-05-19 16:50:45,793 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:50:45,793 Saving new checkpoint.\n",
            "2020-05-19 16:50:46,261 Example #0\n",
            "2020-05-19 16:50:46,262 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:50:46,262 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:50:46,262 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:50:46,262 Example #1\n",
            "2020-05-19 16:50:46,262 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:50:46,262 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:50:46,263 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vhakor .\n",
            "2020-05-19 16:50:46,263 Example #2\n",
            "2020-05-19 16:50:46,263 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:50:46,263 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:50:46,263 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:50:46,263 Example #3\n",
            "2020-05-19 16:50:46,263 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:50:46,264 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:50:46,264 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 16:50:46,264 Validation result (greedy) at epoch  20, step    41000: bleu:  39.50, loss: 29279.4590, ppl:   3.5102, duration: 52.4204s\n",
            "2020-05-19 16:51:34,924 Epoch  20 Step:    41100 Batch Loss:     1.324607 Tokens per Sec:     4509, Lr: 0.000300\n",
            "2020-05-19 16:52:23,904 Epoch  20 Step:    41200 Batch Loss:     1.216721 Tokens per Sec:     4696, Lr: 0.000300\n",
            "2020-05-19 16:53:12,605 Epoch  20 Step:    41300 Batch Loss:     1.247971 Tokens per Sec:     4546, Lr: 0.000300\n",
            "2020-05-19 16:54:01,625 Epoch  20 Step:    41400 Batch Loss:     1.177251 Tokens per Sec:     4675, Lr: 0.000300\n",
            "2020-05-19 16:54:50,373 Epoch  20 Step:    41500 Batch Loss:     1.200280 Tokens per Sec:     4620, Lr: 0.000300\n",
            "2020-05-19 16:55:39,259 Epoch  20 Step:    41600 Batch Loss:     1.300698 Tokens per Sec:     4544, Lr: 0.000300\n",
            "2020-05-19 16:56:27,733 Epoch  20 Step:    41700 Batch Loss:     1.223718 Tokens per Sec:     4557, Lr: 0.000300\n",
            "2020-05-19 16:57:15,998 Epoch  20 Step:    41800 Batch Loss:     1.127809 Tokens per Sec:     4681, Lr: 0.000300\n",
            "2020-05-19 16:58:04,498 Epoch  20 Step:    41900 Batch Loss:     1.238041 Tokens per Sec:     4616, Lr: 0.000300\n",
            "2020-05-19 16:58:52,884 Epoch  20 Step:    42000 Batch Loss:     1.247354 Tokens per Sec:     4509, Lr: 0.000300\n",
            "2020-05-19 16:59:46,218 Hooray! New best validation result [ppl]!\n",
            "2020-05-19 16:59:46,218 Saving new checkpoint.\n",
            "2020-05-19 16:59:46,785 Example #0\n",
            "2020-05-19 16:59:46,785 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 16:59:46,785 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 16:59:46,785 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 16:59:46,785 Example #1\n",
            "2020-05-19 16:59:46,786 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 16:59:46,786 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:59:46,786 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a fhelelwa nga u fhisea kana u vha hawe na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 16:59:46,786 Example #2\n",
            "2020-05-19 16:59:46,786 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 16:59:46,786 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 16:59:46,786 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 16:59:46,787 Example #3\n",
            "2020-05-19 16:59:46,787 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 16:59:46,787 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 16:59:46,787 \tHypothesis: “ Ni ḓo litsha lini u ṱudza kha mavhonele mavhili o fhambanaho ? ” — 1 VHA - KATA 18 : 21 .\n",
            "2020-05-19 16:59:46,787 Validation result (greedy) at epoch  20, step    42000: bleu:  40.09, loss: 29110.0547, ppl:   3.4847, duration: 53.9028s\n",
            "2020-05-19 17:00:22,956 Epoch  20: total training loss 2582.02\n",
            "2020-05-19 17:00:22,956 EPOCH 21\n",
            "2020-05-19 17:00:35,465 Epoch  21 Step:    42100 Batch Loss:     1.153278 Tokens per Sec:     4452, Lr: 0.000300\n",
            "2020-05-19 17:01:23,705 Epoch  21 Step:    42200 Batch Loss:     1.015557 Tokens per Sec:     4542, Lr: 0.000300\n",
            "2020-05-19 17:02:12,266 Epoch  21 Step:    42300 Batch Loss:     1.181850 Tokens per Sec:     4583, Lr: 0.000300\n",
            "2020-05-19 17:03:00,902 Epoch  21 Step:    42400 Batch Loss:     1.190050 Tokens per Sec:     4592, Lr: 0.000300\n",
            "2020-05-19 17:03:48,622 Epoch  21 Step:    42500 Batch Loss:     0.917273 Tokens per Sec:     4542, Lr: 0.000300\n",
            "2020-05-19 17:04:37,672 Epoch  21 Step:    42600 Batch Loss:     1.084407 Tokens per Sec:     4673, Lr: 0.000300\n",
            "2020-05-19 17:05:26,321 Epoch  21 Step:    42700 Batch Loss:     1.203543 Tokens per Sec:     4646, Lr: 0.000300\n",
            "2020-05-19 17:06:15,197 Epoch  21 Step:    42800 Batch Loss:     1.097899 Tokens per Sec:     4675, Lr: 0.000300\n",
            "2020-05-19 17:07:04,012 Epoch  21 Step:    42900 Batch Loss:     1.289672 Tokens per Sec:     4564, Lr: 0.000300\n",
            "2020-05-19 17:07:52,372 Epoch  21 Step:    43000 Batch Loss:     0.938554 Tokens per Sec:     4670, Lr: 0.000300\n",
            "2020-05-19 17:08:54,923 Example #0\n",
            "2020-05-19 17:08:54,923 \tSource:     Jehovah empowered Jesus to perform miracles .\n",
            "2020-05-19 17:08:54,923 \tReference:  Yehova o ṋea Yesu maanḓa a uri a ite vhuṱolo .\n",
            "2020-05-19 17:08:54,923 \tHypothesis: Yehova o ṋea Yesu maanḓa a u ita vhuṱolo .\n",
            "2020-05-19 17:08:54,924 Example #1\n",
            "2020-05-19 17:08:54,924 \tSource:     He endured many hardships , yet he never lost his zeal or his sense of urgency . ​ — 2 Cor .\n",
            "2020-05-19 17:08:54,924 \tReference:  O konḓelela maṱhupho manzhi , fhedzi o bvela phanḓa a tshi fhisea nahone o vha e na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 17:08:54,924 \tHypothesis: O konḓelela vhuleme vhunzhi , fhedzi ho ngo vhuya a litsha u fhisea kana u vha na muya wa u ṱavhanyisa zwithu . — 2 Vha - Kor .\n",
            "2020-05-19 17:08:54,924 Example #2\n",
            "2020-05-19 17:08:54,924 \tSource:     In the first century C.E . , the congregation in Philippi sent Epaphroditus to Rome in order to care for Paul’s physical needs .\n",
            "2020-05-19 17:08:54,924 \tReference:  Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha ngei Filipi tsho rumela Epafrodito ngei Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza lwa ṋama .\n",
            "2020-05-19 17:08:54,925 \tHypothesis: Ḓanani ḽa u thoma ḽa miṅwaha C.E . , tshivhidzo tsha Filipi tsho rumela Epafrodito uri a ye Roma u itela u ṱhogomela ṱhoḓea dza Paulo dza ṋama .\n",
            "2020-05-19 17:08:54,925 Example #3\n",
            "2020-05-19 17:08:54,925 \tSource:     “ How long will you be limping upon two different opinions ? ” ​ — 1 KINGS 18 : 21 .\n",
            "2020-05-19 17:08:54,925 \tReference:  “ Ni ḓo litsha lini u ṱudza thungo dzoṱhe ? ” — 1 DZIKHOSI 18 : 21 .\n",
            "2020-05-19 17:08:54,925 \tHypothesis: “ Ni ḓo guma lini u ṱudza thungo dzoṱhe ? ” — 1 VHA - HEBERU 18 : 21 .\n",
            "2020-05-19 17:08:54,925 Validation result (greedy) at epoch  21, step    43000: bleu:  40.50, loss: 29362.8086, ppl:   3.5227, duration: 62.5526s\n",
            "2020-05-19 17:09:43,725 Epoch  21 Step:    43100 Batch Loss:     1.229031 Tokens per Sec:     4689, Lr: 0.000300\n",
            "2020-05-19 17:10:31,005 Epoch  21 Step:    43200 Batch Loss:     1.346138 Tokens per Sec:     4526, Lr: 0.000300\n",
            "2020-05-19 17:11:19,580 Epoch  21 Step:    43300 Batch Loss:     1.125677 Tokens per Sec:     4625, Lr: 0.000300\n",
            "2020-05-19 17:12:08,599 Epoch  21 Step:    43400 Batch Loss:     1.480299 Tokens per Sec:     4718, Lr: 0.000300\n",
            "2020-05-19 17:12:57,515 Epoch  21 Step:    43500 Batch Loss:     1.322020 Tokens per Sec:     4533, Lr: 0.000300\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n",
            "    \"__main__\", mod_spec)\n",
            "  File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n",
            "    exec(code, run_globals)\n",
            "  File \"/content/joeynmt/joeynmt/__main__.py\", line 41, in <module>\n",
            "    main()\n",
            "  File \"/content/joeynmt/joeynmt/__main__.py\", line 29, in main\n",
            "    train(cfg_file=args.config_path)\n",
            "  File \"/content/joeynmt/joeynmt/training.py\", line 650, in train\n",
            "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
            "  File \"/content/joeynmt/joeynmt/training.py\", line 326, in train_and_validate\n",
            "    batch, update=update, count=count)\n",
            "  File \"/content/joeynmt/joeynmt/training.py\", line 500, in _train_batch\n",
            "    norm_batch_loss.backward()\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/torch/tensor.py\", line 198, in backward\n",
            "    torch.autograd.backward(self, gradient, retain_graph, create_graph)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\", line 100, in backward\n",
            "    allow_unreachable=True)  # allow_unreachable flag\n",
            "KeyboardInterrupt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "6fb95e47-4c65-4acf-9485-3bcd2e412fa6"
      },
      "source": [
        "# Copy the created models from the notebook storage to google drive for persistant storage \n",
        "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\""
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot create symbolic link '/content/drive/My Drive/masakhane/en-ve-baseline/models/enve_transformer/best.ckpt': Operation not supported\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4WeRIo2_66Jq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp /content/joeynmt/models/enve_transformer/best.ckpt \"/content/drive/My Drive/masakhane/en-ve-baseline/models/enve_transformer/\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 759
        },
        "outputId": "eea9edff-0b96-4fb4-ebe0-6aa598692e28"
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 1000\tLoss: 97914.36719\tPPL: 66.62564\tbleu: 0.17632\tLR: 0.00030000\t*\n",
            "Steps: 2000\tLoss: 77647.37500\tPPL: 27.93646\tbleu: 2.44507\tLR: 0.00030000\t*\n",
            "Steps: 3000\tLoss: 66852.96094\tPPL: 17.58438\tbleu: 6.81249\tLR: 0.00030000\t*\n",
            "Steps: 4000\tLoss: 58730.96484\tPPL: 12.41241\tbleu: 10.56772\tLR: 0.00030000\t*\n",
            "Steps: 5000\tLoss: 52724.77344\tPPL: 9.59382\tbleu: 16.37100\tLR: 0.00030000\t*\n",
            "Steps: 6000\tLoss: 48624.95312\tPPL: 8.04698\tbleu: 19.92785\tLR: 0.00030000\t*\n",
            "Steps: 7000\tLoss: 45252.95312\tPPL: 6.96354\tbleu: 23.20015\tLR: 0.00030000\t*\n",
            "Steps: 8000\tLoss: 43076.91016\tPPL: 6.34310\tbleu: 25.56756\tLR: 0.00030000\t*\n",
            "Steps: 9000\tLoss: 41350.00781\tPPL: 5.89031\tbleu: 27.01573\tLR: 0.00030000\t*\n",
            "Steps: 10000\tLoss: 39892.46875\tPPL: 5.53339\tbleu: 28.77102\tLR: 0.00030000\t*\n",
            "Steps: 11000\tLoss: 38838.59766\tPPL: 5.28888\tbleu: 29.92940\tLR: 0.00030000\t*\n",
            "Steps: 12000\tLoss: 37755.30859\tPPL: 5.04879\tbleu: 30.89894\tLR: 0.00030000\t*\n",
            "Steps: 13000\tLoss: 37271.91016\tPPL: 4.94520\tbleu: 32.14625\tLR: 0.00030000\t*\n",
            "Steps: 14000\tLoss: 36745.81641\tPPL: 4.83488\tbleu: 32.01383\tLR: 0.00030000\t*\n",
            "Steps: 15000\tLoss: 35634.11328\tPPL: 4.60978\tbleu: 33.47479\tLR: 0.00030000\t*\n",
            "Steps: 16000\tLoss: 35063.93359\tPPL: 4.49843\tbleu: 34.14596\tLR: 0.00030000\t*\n",
            "Steps: 17000\tLoss: 34512.66797\tPPL: 4.39333\tbleu: 34.14053\tLR: 0.00030000\t*\n",
            "Steps: 18000\tLoss: 34134.87109\tPPL: 4.32272\tbleu: 34.60637\tLR: 0.00030000\t*\n",
            "Steps: 19000\tLoss: 33658.51172\tPPL: 4.23531\tbleu: 35.38740\tLR: 0.00030000\t*\n",
            "Steps: 20000\tLoss: 33328.21875\tPPL: 4.17574\tbleu: 35.77388\tLR: 0.00030000\t*\n",
            "Steps: 21000\tLoss: 33101.50000\tPPL: 4.13534\tbleu: 36.34416\tLR: 0.00030000\t*\n",
            "Steps: 22000\tLoss: 32817.65625\tPPL: 4.08530\tbleu: 35.92540\tLR: 0.00030000\t*\n",
            "Steps: 23000\tLoss: 32373.00195\tPPL: 4.00814\tbleu: 36.44055\tLR: 0.00030000\t*\n",
            "Steps: 24000\tLoss: 32040.88672\tPPL: 3.95145\tbleu: 37.03993\tLR: 0.00030000\t*\n",
            "Steps: 25000\tLoss: 31832.52734\tPPL: 3.91630\tbleu: 37.15437\tLR: 0.00030000\t*\n",
            "Steps: 26000\tLoss: 31805.89453\tPPL: 3.91183\tbleu: 37.51729\tLR: 0.00030000\t*\n",
            "Steps: 27000\tLoss: 31465.85156\tPPL: 3.85520\tbleu: 37.93070\tLR: 0.00030000\t*\n",
            "Steps: 28000\tLoss: 31105.43555\tPPL: 3.79607\tbleu: 37.91469\tLR: 0.00030000\t*\n",
            "Steps: 29000\tLoss: 30879.60742\tPPL: 3.75949\tbleu: 38.02227\tLR: 0.00030000\t*\n",
            "Steps: 30000\tLoss: 30723.86914\tPPL: 3.73446\tbleu: 38.72473\tLR: 0.00030000\t*\n",
            "Steps: 31000\tLoss: 30693.85352\tPPL: 3.72966\tbleu: 38.48172\tLR: 0.00030000\t*\n",
            "Steps: 32000\tLoss: 30646.23047\tPPL: 3.72205\tbleu: 38.90937\tLR: 0.00030000\t*\n",
            "Steps: 33000\tLoss: 30271.55469\tPPL: 3.66272\tbleu: 38.64119\tLR: 0.00030000\t*\n",
            "Steps: 34000\tLoss: 30388.94531\tPPL: 3.68120\tbleu: 38.35433\tLR: 0.00030000\t\n",
            "Steps: 35000\tLoss: 30051.58594\tPPL: 3.62833\tbleu: 39.18020\tLR: 0.00030000\t*\n",
            "Steps: 36000\tLoss: 29892.06250\tPPL: 3.60359\tbleu: 39.35048\tLR: 0.00030000\t*\n",
            "Steps: 37000\tLoss: 29705.04492\tPPL: 3.57480\tbleu: 39.44941\tLR: 0.00030000\t*\n",
            "Steps: 38000\tLoss: 29640.87695\tPPL: 3.56498\tbleu: 39.82367\tLR: 0.00030000\t*\n",
            "Steps: 39000\tLoss: 29443.34961\tPPL: 3.53491\tbleu: 39.54081\tLR: 0.00030000\t*\n",
            "Steps: 40000\tLoss: 29347.57422\tPPL: 3.52042\tbleu: 39.89243\tLR: 0.00030000\t*\n",
            "Steps: 41000\tLoss: 29279.45898\tPPL: 3.51015\tbleu: 39.50134\tLR: 0.00030000\t*\n",
            "Steps: 42000\tLoss: 29110.05469\tPPL: 3.48474\tbleu: 40.09332\tLR: 0.00030000\t*\n",
            "Steps: 43000\tLoss: 29362.80859\tPPL: 3.52272\tbleu: 40.50369\tLR: 0.00030000\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "6cb32714-3d0e-4f42-aae7-faf5c80c42b2"
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-05-19 17:16:18,491 Hello! This is Joey-NMT.\n",
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n",
            "2020-05-19 17:17:38,145  dev bleu:  41.14 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-05-19 17:19:33,397 test bleu:  49.57 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "foeaIXVo75Pe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}