en-ish/jw300-baseline/English_to_Esan_Word-level_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "starter_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ruohoruotsi/masakhane/blob/master/en-ish/jw300-baseline/English_to_Esan_Word-level_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "2a02d3ea-3f1a-46b9-bd56-f439adb91530",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"ish\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "b79d00f5-b467-4493-cd76-26dc880ca2bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-ish-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "85f1a7de-5ab8-45b9-cdb0-9ae3199df91b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 23.8MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 1.7MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 1.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 2.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 2.9MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 2.5MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "outputId": "9d17c9e1-b6a7-43f4-ee1e-3ea9a8559c27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-iso.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            "   2 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-iso.xml.gz\n",
            " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
            "  26 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/iso.zip\n",
            "\n",
            " 291 MB Total size\n",
            "./JW300_latest_xml_en-iso.xml.gz ... 100% of 2 MB\n",
            "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
            "./JW300_latest_xml_iso.zip ... 100% of 26 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "outputId": "8949fb1b-854c-4644-a03c-43fdb68dbc42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-01-17 04:59:50--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2020-01-17 04:59:51 (5.15 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2020-01-17 04:59:52--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-iso.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 205488 (201K) [text/plain]\n",
            "Saving to: ‘test.en-iso.en’\n",
            "\n",
            "test.en-iso.en      100%[===================>] 200.67K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2020-01-17 04:59:52 (5.30 MB/s) - ‘test.en-iso.en’ saved [205488/205488]\n",
            "\n",
            "--2020-01-17 04:59:54--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-iso.iso\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 251918 (246K) [text/plain]\n",
            "Saving to: ‘test.en-iso.iso’\n",
            "\n",
            "test.en-iso.iso     100%[===================>] 246.01K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2020-01-17 04:59:55 (4.68 MB/s) - ‘test.en-iso.iso’ saved [251918/251918]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "outputId": "44ef9271-b0fc-418d-dc5f-04c90e4e38cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "fcf09810-26de-481b-8ba0-39853c4378b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 5685/243487 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>“ Only One of the Many Lives That You Touched ”</td>\n",
              "      <td>“ Omọvo Ahwo Buobu nọ Who Duobọ te Uzuazọ Riẹ ”</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>BACK in January 1996 , Carol was sick with a b...</td>\n",
              "      <td>EVAỌ January 1996 , Carol ọ jẹ mọ ẹyao ẹvori .</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>She was in her 60 ’ s and until then had alway...</td>\n",
              "      <td>Ọ kpako te ikpe 60 no yọ oke yena kpobi ọ jọ a...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0    “ Only One of the Many Lives That You Touched ”    “ Omọvo Ahwo Buobu nọ Who Duobọ te Uzuazọ Riẹ ”\n",
              "1  BACK in January 1996 , Carol was sick with a b...     EVAỌ January 1996 , Carol ọ jẹ mọ ẹyao ẹvori .\n",
              "2  She was in her 60 ’ s and until then had alway...  Ọ kpako te ikpe 60 no yọ oke yena kpobi ọ jọ a..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "b109ef42-7557-4c8c-8952-874ac33ed667",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "outputId": "175b8c38-2c43-47f7-bccc-0cffe0336959",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.17.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 1.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (42.0.2)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144672 sha256=27cc679925e9ea7d499147fe08f2564ad8e4295a570a1a075b46139a669d35bc\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n",
            "00:00:00.10 0.00 percent complete\n",
            "00:00:23.78 0.46 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:00:47.40 0.92 percent complete\n",
            "00:01:10.57 1.39 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:01:33.67 1.85 percent complete\n",
            "00:01:56.94 2.31 percent complete\n",
            "00:02:20.72 2.77 percent complete\n",
            "00:02:43.97 3.23 percent complete\n",
            "00:03:07.38 3.70 percent complete\n",
            "00:03:31.62 4.16 percent complete\n",
            "00:03:56.02 4.62 percent complete\n",
            "00:04:19.06 5.08 percent complete\n",
            "00:04:42.39 5.54 percent complete\n",
            "00:05:06.25 6.00 percent complete\n",
            "00:05:30.46 6.47 percent complete\n",
            "00:05:53.63 6.93 percent complete\n",
            "00:06:17.23 7.39 percent complete\n",
            "00:06:41.67 7.85 percent complete\n",
            "00:07:05.22 8.31 percent complete\n",
            "00:07:29.08 8.78 percent complete\n",
            "00:07:52.78 9.24 percent complete\n",
            "00:08:16.19 9.70 percent complete\n",
            "00:08:39.44 10.16 percent complete\n",
            "00:09:03.31 10.62 percent complete\n",
            "00:09:26.40 11.09 percent complete\n",
            "00:09:51.25 11.55 percent complete\n",
            "00:10:15.14 12.01 percent complete\n",
            "00:10:38.52 12.47 percent complete\n",
            "00:11:02.41 12.93 percent complete\n",
            "00:11:26.41 13.40 percent complete\n",
            "00:11:50.05 13.86 percent complete\n",
            "00:12:13.38 14.32 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:12:37.46 14.78 percent complete\n",
            "00:13:01.81 15.24 percent complete\n",
            "00:13:25.69 15.70 percent complete\n",
            "00:13:49.63 16.17 percent complete\n",
            "00:14:13.48 16.63 percent complete\n",
            "00:14:37.29 17.09 percent complete\n",
            "00:15:01.50 17.55 percent complete\n",
            "00:15:25.57 18.01 percent complete\n",
            "00:15:49.18 18.48 percent complete\n",
            "00:16:13.51 18.94 percent complete\n",
            "00:16:37.01 19.40 percent complete\n",
            "00:17:00.10 19.86 percent complete\n",
            "00:17:23.41 20.32 percent complete\n",
            "00:17:47.00 20.79 percent complete\n",
            "00:18:10.96 21.25 percent complete\n",
            "00:18:34.67 21.71 percent complete\n",
            "00:18:59.32 22.17 percent complete\n",
            "00:19:23.61 22.63 percent complete\n",
            "00:19:47.81 23.10 percent complete\n",
            "00:20:12.35 23.56 percent complete\n",
            "00:20:36.05 24.02 percent complete\n",
            "00:20:59.50 24.48 percent complete\n",
            "00:21:23.22 24.94 percent complete\n",
            "00:21:47.62 25.40 percent complete\n",
            "00:22:11.51 25.87 percent complete\n",
            "00:22:35.28 26.33 percent complete\n",
            "00:22:58.71 26.79 percent complete\n",
            "00:23:22.48 27.25 percent complete\n",
            "00:23:46.81 27.71 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:24:10.94 28.18 percent complete\n",
            "00:24:34.16 28.64 percent complete\n",
            "00:24:58.34 29.10 percent complete\n",
            "00:25:22.14 29.56 percent complete\n",
            "00:25:46.52 30.02 percent complete\n",
            "00:26:09.90 30.49 percent complete\n",
            "00:26:34.10 30.95 percent complete\n",
            "00:26:58.06 31.41 percent complete\n",
            "00:27:21.26 31.87 percent complete\n",
            "00:27:45.50 32.33 percent complete\n",
            "00:28:09.00 32.79 percent complete\n",
            "00:28:32.87 33.26 percent complete\n",
            "00:28:57.11 33.72 percent complete\n",
            "00:29:21.37 34.18 percent complete\n",
            "00:29:45.41 34.64 percent complete\n",
            "00:30:08.93 35.10 percent complete\n",
            "00:30:32.97 35.57 percent complete\n",
            "00:30:56.46 36.03 percent complete\n",
            "00:31:19.93 36.49 percent complete\n",
            "00:31:44.33 36.95 percent complete\n",
            "00:32:07.88 37.41 percent complete\n",
            "00:32:32.08 37.88 percent complete\n",
            "00:32:56.34 38.34 percent complete\n",
            "00:33:20.79 38.80 percent complete\n",
            "00:33:44.59 39.26 percent complete\n",
            "00:34:08.98 39.72 percent complete\n",
            "00:34:32.93 40.19 percent complete\n",
            "00:34:56.72 40.65 percent complete\n",
            "00:35:21.12 41.11 percent complete\n",
            "00:35:45.38 41.57 percent complete\n",
            "00:36:09.10 42.03 percent complete\n",
            "00:36:33.30 42.49 percent complete\n",
            "00:36:57.68 42.96 percent complete\n",
            "00:37:21.90 43.42 percent complete\n",
            "00:37:45.21 43.88 percent complete\n",
            "00:38:09.01 44.34 percent complete\n",
            "00:38:33.14 44.80 percent complete\n",
            "00:38:57.65 45.27 percent complete\n",
            "00:39:21.34 45.73 percent complete\n",
            "00:39:45.27 46.19 percent complete\n",
            "00:40:09.16 46.65 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:40:32.51 47.11 percent complete\n",
            "00:40:56.64 47.58 percent complete\n",
            "00:41:20.36 48.04 percent complete\n",
            "00:41:44.61 48.50 percent complete\n",
            "00:42:08.32 48.96 percent complete\n",
            "00:42:31.93 49.42 percent complete\n",
            "00:42:56.28 49.89 percent complete\n",
            "00:43:20.24 50.35 percent complete\n",
            "00:43:44.62 50.81 percent complete\n",
            "00:44:08.91 51.27 percent complete\n",
            "00:44:32.54 51.73 percent complete\n",
            "00:44:56.09 52.19 percent complete\n",
            "00:45:20.95 52.66 percent complete\n",
            "00:45:44.71 53.12 percent complete\n",
            "00:46:08.30 53.58 percent complete\n",
            "00:46:32.61 54.04 percent complete\n",
            "00:46:56.86 54.50 percent complete\n",
            "00:47:20.82 54.97 percent complete\n",
            "00:47:44.99 55.43 percent complete\n",
            "00:48:08.38 55.89 percent complete\n",
            "00:48:32.48 56.35 percent complete\n",
            "00:48:56.67 56.81 percent complete\n",
            "00:49:20.35 57.28 percent complete\n",
            "00:49:44.61 57.74 percent complete\n",
            "00:50:08.07 58.20 percent complete\n",
            "00:50:32.40 58.66 percent complete\n",
            "00:50:56.51 59.12 percent complete\n",
            "00:51:20.65 59.59 percent complete\n",
            "00:51:44.56 60.05 percent complete\n",
            "00:52:08.95 60.51 percent complete\n",
            "00:52:32.77 60.97 percent complete\n",
            "00:52:56.32 61.43 percent complete\n",
            "00:53:20.96 61.89 percent complete\n",
            "00:53:45.01 62.36 percent complete\n",
            "00:54:08.62 62.82 percent complete\n",
            "00:54:32.80 63.28 percent complete\n",
            "00:54:56.52 63.74 percent complete\n",
            "00:55:20.42 64.20 percent complete\n",
            "00:55:44.48 64.67 percent complete\n",
            "00:56:08.38 65.13 percent complete\n",
            "00:56:32.50 65.59 percent complete\n",
            "00:56:56.41 66.05 percent complete\n",
            "00:57:20.00 66.51 percent complete\n",
            "00:57:43.52 66.98 percent complete\n",
            "00:58:07.51 67.44 percent complete\n",
            "00:58:31.43 67.90 percent complete\n",
            "00:58:55.35 68.36 percent complete\n",
            "00:59:19.06 68.82 percent complete\n",
            "00:59:42.85 69.29 percent complete\n",
            "01:00:07.00 69.75 percent complete\n",
            "01:00:30.53 70.21 percent complete\n",
            "01:00:54.45 70.67 percent complete\n",
            "01:01:18.35 71.13 percent complete\n",
            "01:01:42.37 71.59 percent complete\n",
            "01:02:06.05 72.06 percent complete\n",
            "01:02:29.54 72.52 percent complete\n",
            "01:02:53.79 72.98 percent complete\n",
            "01:03:17.95 73.44 percent complete\n",
            "01:03:41.78 73.90 percent complete\n",
            "01:04:05.36 74.37 percent complete\n",
            "01:04:29.49 74.83 percent complete\n",
            "01:04:54.40 75.29 percent complete\n",
            "01:05:18.19 75.75 percent complete\n",
            "01:05:41.97 76.21 percent complete\n",
            "01:06:05.42 76.68 percent complete\n",
            "01:06:29.65 77.14 percent complete\n",
            "01:06:53.24 77.60 percent complete\n",
            "01:07:16.62 78.06 percent complete\n",
            "01:07:40.89 78.52 percent complete\n",
            "01:08:05.22 78.98 percent complete\n",
            "01:08:28.49 79.45 percent complete\n",
            "01:08:52.56 79.91 percent complete\n",
            "01:09:16.44 80.37 percent complete\n",
            "01:09:40.34 80.83 percent complete\n",
            "01:10:04.15 81.29 percent complete\n",
            "01:10:28.28 81.76 percent complete\n",
            "01:10:51.83 82.22 percent complete\n",
            "01:11:15.87 82.68 percent complete\n",
            "01:11:39.95 83.14 percent complete\n",
            "01:12:03.72 83.60 percent complete\n",
            "01:12:26.92 84.07 percent complete\n",
            "01:12:51.37 84.53 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇩']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:13:15.72 84.99 percent complete\n",
            "01:13:39.67 85.45 percent complete\n",
            "01:14:03.39 85.91 percent complete\n",
            "01:14:27.32 86.38 percent complete\n",
            "01:14:51.76 86.84 percent complete\n",
            "01:15:15.58 87.30 percent complete\n",
            "01:15:39.33 87.76 percent complete\n",
            "01:16:03.85 88.22 percent complete\n",
            "01:16:27.71 88.68 percent complete\n",
            "01:16:51.58 89.15 percent complete\n",
            "01:17:15.41 89.61 percent complete\n",
            "01:17:39.83 90.07 percent complete\n",
            "01:18:04.13 90.53 percent complete\n",
            "01:18:28.67 90.99 percent complete\n",
            "01:18:53.01 91.46 percent complete\n",
            "01:19:16.73 91.92 percent complete\n",
            "01:19:41.87 92.38 percent complete\n",
            "01:20:05.67 92.84 percent complete\n",
            "01:20:29.44 93.30 percent complete\n",
            "01:20:53.86 93.77 percent complete\n",
            "01:21:18.33 94.23 percent complete\n",
            "01:21:42.36 94.69 percent complete\n",
            "01:22:06.26 95.15 percent complete\n",
            "01:22:29.82 95.61 percent complete\n",
            "01:22:53.83 96.08 percent complete\n",
            "01:23:17.51 96.54 percent complete\n",
            "01:23:41.70 97.00 percent complete\n",
            "01:24:05.69 97.46 percent complete\n",
            "01:24:30.08 97.92 percent complete\n",
            "01:24:53.94 98.38 percent complete\n",
            "01:25:17.57 98.85 percent complete\n",
            "01:25:41.43 99.31 percent complete\n",
            "01:26:05.41 99.77 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "colab": {}
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "outputId": "bff98114-48cb-4762-9a92-c29167222bbc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 61, done.\u001b[K\n",
            "remote: Counting objects: 100% (61/61), done.\u001b[K\n",
            "remote: Compressing objects: 100% (39/39), done.\u001b[K\n",
            "remote: Total 2245 (delta 34), reused 34 (delta 22), pack-reused 2184\u001b[K\n",
            "Receiving objects: 100% (2245/2245), 2.63 MiB | 1.95 MiB/s, done.\n",
            "Resolving deltas: 100% (1555/1555), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (6.2.2)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (42.0.2)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "  Downloading https://files.pythonhosted.org/packages/45/31/1a135b964c169984b27fb2f7a50280fa7f8e6d9d404d8a9e596180487fd1/sacrebleu-1.4.3-py3-none-any.whl\n",
            "Collecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.2)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 34.6MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
            "\u001b[K     |████████████████████████████████| 307kB 50.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.6)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.4.1)\n",
            "Collecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 9.4MB/s \n",
            "\u001b[?25hCollecting astroid<2.4,>=2.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 53.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.11.28)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 55.9MB/s \n",
            "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 10.3MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=0dd2eb8283fc02915ba7744edd0efbcfa015c5bde8a28d89e2a61f6bda0083bc\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-c6jfu0up/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=44229 sha256=df87aadcc50a400dc278a40afc1ffa777492a692efd93def19fcb776e5656e14\n",
            "  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191\n",
            "Successfully built joeynmt pyyaml\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, mccabe, isort, typed-ast, lazy-object-proxy, astroid, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.3 sacrebleu-1.4.3 subword-nmt-0.3.7 typed-ast-1.4.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "colab": {}
      },
      "source": [
        "# # One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# # Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# # Do subword NMT\n",
        "# from os import path\n",
        "# os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "# os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# # Learn BPEs on the training data.\n",
        "# os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "# ! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# # Apply BPE splits to the development and test data.\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "# ! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# # Create directory, move everyone we care about to the correct location\n",
        "# ! mkdir -p $data_path\n",
        "# ! cp train.* $data_path\n",
        "# ! cp test.* $data_path\n",
        "# ! cp dev.* $data_path\n",
        "# ! cp bpe.codes.4000 $data_path\n",
        "# ! ls $data_path\n",
        "\n",
        "# # Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "# ! cp train.* \"$gdrive_path\"\n",
        "# ! cp test.* \"$gdrive_path\"\n",
        "# ! cp dev.* \"$gdrive_path\"\n",
        "# ! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "# ! ls \"$gdrive_path\"\n",
        "\n",
        "# # Create that vocab using build_vocab\n",
        "# ! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "# ! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# # Some output\n",
        "# ! echo \"BPE Isoko Sentences\"\n",
        "# ! tail -n 5 test.bpe.$tgt\n",
        "# ! echo \"Combined BPE Vocab\"\n",
        "# ! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman\n",
        "# ! cp joeynmt/data/$src$tgt/vocab.txt \"$gdrive_path\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4OkAKSIK7Eg4",
        "colab_type": "code",
        "outputId": "51b9e870-a6cd-4089-f92f-427a5bc3a43d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "###### IOHAVOC MODIFICATIONS ==>> CREATE THE VOCAB FOR NON-BPE EXPERIMENTS\n",
        "from os import path\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py \"$gdrive_path/train.$src\" \"$gdrive_path/train.$tgt\" --output_path \"$gdrive_path/vocab-nonBPE.txt\"\n",
        "\n",
        "# Some output\n",
        "! echo \"Ẹ̀sán test Sentences\"\n",
        "! tail -n 5 \"$gdrive_path/test.$tgt\"\n",
        "! echo \"Combined Vocab\"\n",
        "! tail -n 10 \"$gdrive_path/vocab-nonBPE.txt\"  # iroro"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Ẹ̀sán test Sentences\n",
            "Agbada ọsi ẹlinmhin nọn khiale ( Fẹ uduọle 19 - 20 ghe )\n",
            "Jehova dẹ rẹkpa mhan nin mhan da sabọ nin ọle suan .\n",
            "( b ) Emhin nela mhan ha zilo nyan bhi uhọmhọn - ọta nọn ki sẹ ọle bhi egbe ?\n",
            "Uwẹ be gbẹlokotọ tie ene ebe Ọkhẹughe nọn bha sẹ buẹ gbe nan ne dagbare ?\n",
            "Fẹghe si uwẹ dẹ sabọ re ewanniẹn ọbhi ene inọnta nan :\n",
            "Combined Vocab\n",
            "reassured\n",
            "devote\n",
            "thrill\n",
            "commendable\n",
            "worthwhile\n",
            "loyally\n",
            "testifying\n",
            "perishes\n",
            "opposite\n",
            "inactive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"{gdrive_path}/train\"\n",
        "    dev:   \"{gdrive_path}/dev\"\n",
        "    test:  \"{gdrive_path}/test\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"{gdrive_path}/vocab-nonBPE.txt\"\n",
        "    trg_vocab: \"{gdrive_path}/vocab-nonBPE.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    # load_model: \"{gdrive_path}/models/{name}_transformer_orig/142000.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 120                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 100          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"{gdrive_path}/models/{name}_transformer\"\n",
        "    overwrite: True               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)\n",
        "\n",
        "! cp joeynmt/configs/transformer_$src$tgt.yaml \"$gdrive_path\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "outputId": "b5c806b5-e0f5-4d40-d5ab-f1d019a0b289",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "# !cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml\n",
        "!python3 -m joeynmt train \"$gdrive_path/transformer_$src$tgt.yaml\""
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-01-26 07:56:54,581 Hello! This is Joey-NMT.\n",
            "2020-01-26 07:56:56,002 Total params: 13244928\n",
            "2020-01-26 07:56:56,003 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2020-01-26 07:57:05,950 cfg.name                           : enish_transformer\n",
            "2020-01-26 07:57:05,950 cfg.data.src                       : en\n",
            "2020-01-26 07:57:05,950 cfg.data.trg                       : ish\n",
            "2020-01-26 07:57:05,950 cfg.data.train                     : /content/drive/My Drive/masakhane/en-ish-baseline/train\n",
            "2020-01-26 07:57:05,950 cfg.data.dev                       : /content/drive/My Drive/masakhane/en-ish-baseline/dev\n",
            "2020-01-26 07:57:05,950 cfg.data.test                      : /content/drive/My Drive/masakhane/en-ish-baseline/test\n",
            "2020-01-26 07:57:05,951 cfg.data.level                     : bpe\n",
            "2020-01-26 07:57:05,951 cfg.data.lowercase                 : False\n",
            "2020-01-26 07:57:05,951 cfg.data.max_sent_length           : 100\n",
            "2020-01-26 07:57:05,951 cfg.data.src_vocab                 : /content/drive/My Drive/masakhane/en-ish-baseline/vocab-nonBPE.txt\n",
            "2020-01-26 07:57:05,951 cfg.data.trg_vocab                 : /content/drive/My Drive/masakhane/en-ish-baseline/vocab-nonBPE.txt\n",
            "2020-01-26 07:57:05,951 cfg.testing.beam_size              : 5\n",
            "2020-01-26 07:57:05,951 cfg.testing.alpha                  : 1.0\n",
            "2020-01-26 07:57:05,951 cfg.training.random_seed           : 42\n",
            "2020-01-26 07:57:05,952 cfg.training.optimizer             : adam\n",
            "2020-01-26 07:57:05,952 cfg.training.normalization         : tokens\n",
            "2020-01-26 07:57:05,952 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2020-01-26 07:57:05,952 cfg.training.scheduling            : plateau\n",
            "2020-01-26 07:57:05,952 cfg.training.patience              : 5\n",
            "2020-01-26 07:57:05,952 cfg.training.learning_rate_factor  : 0.5\n",
            "2020-01-26 07:57:05,952 cfg.training.learning_rate_warmup  : 1000\n",
            "2020-01-26 07:57:05,953 cfg.training.decrease_factor       : 0.7\n",
            "2020-01-26 07:57:05,953 cfg.training.loss                  : crossentropy\n",
            "2020-01-26 07:57:05,953 cfg.training.learning_rate         : 0.0003\n",
            "2020-01-26 07:57:05,953 cfg.training.learning_rate_min     : 1e-08\n",
            "2020-01-26 07:57:05,953 cfg.training.weight_decay          : 0.0\n",
            "2020-01-26 07:57:05,953 cfg.training.label_smoothing       : 0.1\n",
            "2020-01-26 07:57:05,953 cfg.training.batch_size            : 4096\n",
            "2020-01-26 07:57:05,954 cfg.training.batch_type            : token\n",
            "2020-01-26 07:57:05,954 cfg.training.eval_batch_size       : 3600\n",
            "2020-01-26 07:57:05,954 cfg.training.eval_batch_type       : token\n",
            "2020-01-26 07:57:05,954 cfg.training.batch_multiplier      : 1\n",
            "2020-01-26 07:57:05,954 cfg.training.early_stopping_metric : ppl\n",
            "2020-01-26 07:57:05,954 cfg.training.epochs                : 120\n",
            "2020-01-26 07:57:05,955 cfg.training.validation_freq       : 100\n",
            "2020-01-26 07:57:05,955 cfg.training.logging_freq          : 100\n",
            "2020-01-26 07:57:05,955 cfg.training.eval_metric           : bleu\n",
            "2020-01-26 07:57:05,955 cfg.training.model_dir             : /content/drive/My Drive/masakhane/en-ish-baseline/models/enish_transformer\n",
            "2020-01-26 07:57:05,955 cfg.training.overwrite             : True\n",
            "2020-01-26 07:57:05,955 cfg.training.shuffle               : True\n",
            "2020-01-26 07:57:05,956 cfg.training.use_cuda              : True\n",
            "2020-01-26 07:57:05,956 cfg.training.max_output_length     : 100\n",
            "2020-01-26 07:57:05,956 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2020-01-26 07:57:05,956 cfg.training.keep_last_ckpts       : 3\n",
            "2020-01-26 07:57:05,956 cfg.model.initializer              : xavier\n",
            "2020-01-26 07:57:05,956 cfg.model.bias_initializer         : zeros\n",
            "2020-01-26 07:57:05,957 cfg.model.init_gain                : 1.0\n",
            "2020-01-26 07:57:05,957 cfg.model.embed_initializer        : xavier\n",
            "2020-01-26 07:57:05,957 cfg.model.embed_init_gain          : 1.0\n",
            "2020-01-26 07:57:05,957 cfg.model.tied_embeddings          : True\n",
            "2020-01-26 07:57:05,957 cfg.model.tied_softmax             : True\n",
            "2020-01-26 07:57:05,957 cfg.model.encoder.type             : transformer\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.num_layers       : 6\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.num_heads        : 4\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.embeddings.scale : True\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.hidden_size      : 256\n",
            "2020-01-26 07:57:05,958 cfg.model.encoder.ff_size          : 1024\n",
            "2020-01-26 07:57:05,959 cfg.model.encoder.dropout          : 0.3\n",
            "2020-01-26 07:57:05,959 cfg.model.decoder.type             : transformer\n",
            "2020-01-26 07:57:05,959 cfg.model.decoder.num_layers       : 6\n",
            "2020-01-26 07:57:05,959 cfg.model.decoder.num_heads        : 4\n",
            "2020-01-26 07:57:05,959 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2020-01-26 07:57:05,959 cfg.model.decoder.embeddings.scale : True\n",
            "2020-01-26 07:57:05,960 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2020-01-26 07:57:05,960 cfg.model.decoder.hidden_size      : 256\n",
            "2020-01-26 07:57:05,960 cfg.model.decoder.ff_size          : 1024\n",
            "2020-01-26 07:57:05,960 cfg.model.decoder.dropout          : 0.3\n",
            "2020-01-26 07:57:05,960 Data set sizes: \n",
            "\ttrain 4126,\n",
            "\tvalid 1000,\n",
            "\ttest 343\n",
            "2020-01-26 07:57:05,960 First training example:\n",
            "\t[SRC] Their unity would give a powerful witness , offering clear evidence that Jehovah had sent Jesus to the earth to do God’s will .\n",
            "\t[TRG] Okugbe nọnsele dẹ wo sabọ sọsali ọbhọ ghe , Jehova gene ji Jesu re bhi ọne otọ nan nin ọle dọ lu iho nọnsole . Ahoẹmhọn - egbe hi iyaman nan ha rẹ gene lẹn edibo nesi Jesu .\n",
            "2020-01-26 07:57:05,961 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) nin (7) ha (8) the (9) mhan\n",
            "2020-01-26 07:57:05,961 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) nin (7) ha (8) the (9) mhan\n",
            "2020-01-26 07:57:05,961 Number of Src words (types): 8534\n",
            "2020-01-26 07:57:05,962 Number of Trg words (types): 8534\n",
            "2020-01-26 07:57:05,962 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=8534),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=8534))\n",
            "2020-01-26 07:57:05,975 EPOCH 1\n",
            "2020-01-26 07:57:12,898 Epoch   1: total training loss 273.35\n",
            "2020-01-26 07:57:12,898 EPOCH 2\n",
            "2020-01-26 07:57:19,618 Epoch   2: total training loss 222.69\n",
            "2020-01-26 07:57:19,619 EPOCH 3\n",
            "2020-01-26 07:57:21,144 Epoch   3 Step:      100 Batch Loss:     4.997618 Tokens per Sec:    13210, Lr: 0.000300\n",
            "2020-01-26 07:58:03,963 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 07:58:03,963 Saving new checkpoint.\n",
            "2020-01-26 07:58:05,059 Example #0\n",
            "2020-01-26 07:58:05,060 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 07:58:05,060 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 07:58:05,060 \tHypothesis: , , , , , , .\n",
            "2020-01-26 07:58:05,060 Example #1\n",
            "2020-01-26 07:58:05,061 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 07:58:05,061 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 07:58:05,061 \tHypothesis: , , , , , , , ,\n",
            "2020-01-26 07:58:05,061 Example #2\n",
            "2020-01-26 07:58:05,062 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 07:58:05,062 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 07:58:05,062 \tHypothesis: , , , , , , , ,\n",
            "2020-01-26 07:58:05,062 Example #3\n",
            "2020-01-26 07:58:05,062 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 07:58:05,063 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 07:58:05,063 \tHypothesis: , , , , , , ,\n",
            "2020-01-26 07:58:05,063 Validation result (greedy) at epoch   3, step      100: bleu:   0.00, loss: 110740.5469, ppl: 135.4943, duration: 43.9188s\n",
            "2020-01-26 07:58:10,428 Epoch   3: total training loss 219.20\n",
            "2020-01-26 07:58:10,429 EPOCH 4\n",
            "2020-01-26 07:58:17,485 Epoch   4: total training loss 211.90\n",
            "2020-01-26 07:58:17,485 EPOCH 5\n",
            "2020-01-26 07:58:20,481 Epoch   5 Step:      200 Batch Loss:     4.402267 Tokens per Sec:    13032, Lr: 0.000300\n",
            "2020-01-26 07:59:04,482 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 07:59:04,482 Saving new checkpoint.\n",
            "2020-01-26 07:59:05,588 Example #0\n",
            "2020-01-26 07:59:05,588 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 07:59:05,588 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 07:59:05,589 \tHypothesis: ( : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :\n",
            "2020-01-26 07:59:05,589 Example #1\n",
            "2020-01-26 07:59:05,589 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 07:59:05,589 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 07:59:05,589 \tHypothesis: ( : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :\n",
            "2020-01-26 07:59:05,589 Example #2\n",
            "2020-01-26 07:59:05,590 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 07:59:05,590 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 07:59:05,590 \tHypothesis: ( : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :\n",
            "2020-01-26 07:59:05,590 Example #3\n",
            "2020-01-26 07:59:05,590 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 07:59:05,590 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 07:59:05,590 \tHypothesis: ( : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :\n",
            "2020-01-26 07:59:05,590 Validation result (greedy) at epoch   5, step      200: bleu:   0.00, loss: 103501.9688, ppl:  98.3032, duration: 45.1093s\n",
            "2020-01-26 07:59:09,498 Epoch   5: total training loss 205.32\n",
            "2020-01-26 07:59:09,498 EPOCH 6\n",
            "2020-01-26 07:59:16,666 Epoch   6: total training loss 200.77\n",
            "2020-01-26 07:59:16,666 EPOCH 7\n",
            "2020-01-26 07:59:21,318 Epoch   7 Step:      300 Batch Loss:     4.303302 Tokens per Sec:    13440, Lr: 0.000300\n",
            "2020-01-26 08:00:06,391 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:00:06,392 Saving new checkpoint.\n",
            "2020-01-26 08:00:07,987 Example #0\n",
            "2020-01-26 08:00:07,988 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:00:07,988 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:00:07,988 \tHypothesis: Be mhan ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha\n",
            "2020-01-26 08:00:07,988 Example #1\n",
            "2020-01-26 08:00:07,989 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:00:07,989 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:00:07,989 \tHypothesis: ( : ( ) ( ) ( ) ( ) ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 08:00:07,989 Example #2\n",
            "2020-01-26 08:00:07,990 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:00:07,990 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:00:07,990 \tHypothesis: ( : ( ) ( ) ( ) ( ) ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 08:00:07,990 Example #3\n",
            "2020-01-26 08:00:07,991 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:00:07,991 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:00:07,991 \tHypothesis: ( : Be , : Be , : Be , mhan ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha\n",
            "2020-01-26 08:00:07,991 Validation result (greedy) at epoch   7, step      300: bleu:   0.00, loss: 96958.1016, ppl:  73.5509, duration: 46.6734s\n",
            "2020-01-26 08:00:10,417 Epoch   7: total training loss 194.45\n",
            "2020-01-26 08:00:10,418 EPOCH 8\n",
            "2020-01-26 08:00:17,533 Epoch   8: total training loss 187.50\n",
            "2020-01-26 08:00:17,533 EPOCH 9\n",
            "2020-01-26 08:00:23,991 Epoch   9 Step:      400 Batch Loss:     4.006420 Tokens per Sec:    12786, Lr: 0.000300\n",
            "2020-01-26 08:01:09,696 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:01:09,696 Saving new checkpoint.\n",
            "2020-01-26 08:01:10,867 Example #0\n",
            "2020-01-26 08:01:10,868 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:01:10,868 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:01:10,868 \tHypothesis: ( : : 1 : 1 : 1 : 1 : (\n",
            "2020-01-26 08:01:10,868 Example #1\n",
            "2020-01-26 08:01:10,869 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:01:10,869 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:01:10,869 \tHypothesis: ( : : 1 ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( 1 ) ( ) ( 1 ) ( ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( ( ( ( ( ( ( ( 1 ) ( 1 ) ( 1\n",
            "2020-01-26 08:01:10,869 Example #2\n",
            "2020-01-26 08:01:10,870 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:01:10,870 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:01:10,870 \tHypothesis: ( ) ( ) Bhi , mhan ha ha ha ha ha ha ha ha ha ha ha ha ha ha re emhin nin mhan ha ha ha ha ha ha re emhin nin mhan ha re emhin nin mhan ha re egbe bhi egbe .\n",
            "2020-01-26 08:01:10,870 Example #3\n",
            "2020-01-26 08:01:10,871 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:01:10,871 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:01:10,871 \tHypothesis: ( : : 1 ) ( ) Be ha ha ha ha ha ha ha re egbe .\n",
            "2020-01-26 08:01:10,872 Validation result (greedy) at epoch   9, step      400: bleu:   0.17, loss: 90492.0625, ppl:  55.2214, duration: 46.8799s\n",
            "2020-01-26 08:01:11,652 Epoch   9: total training loss 181.92\n",
            "2020-01-26 08:01:11,652 EPOCH 10\n",
            "2020-01-26 08:01:18,998 Epoch  10: total training loss 176.76\n",
            "2020-01-26 08:01:18,998 EPOCH 11\n",
            "2020-01-26 08:01:26,067 Epoch  11: total training loss 171.99\n",
            "2020-01-26 08:01:26,068 EPOCH 12\n",
            "2020-01-26 08:01:26,855 Epoch  12 Step:      500 Batch Loss:     3.999224 Tokens per Sec:    12500, Lr: 0.000300\n",
            "2020-01-26 08:02:12,940 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:02:12,940 Saving new checkpoint.\n",
            "2020-01-26 08:02:14,106 Example #0\n",
            "2020-01-26 08:02:14,107 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:02:14,107 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:02:14,107 \tHypothesis: Bhi , ọle da ha ha ha re egbe .\n",
            "2020-01-26 08:02:14,107 Example #1\n",
            "2020-01-26 08:02:14,107 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:02:14,108 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:02:14,108 \tHypothesis: ( 1 : 1 ) Be ha ha ha ha ha ha ha ha ha ha re egbe .\n",
            "2020-01-26 08:02:14,108 Example #2\n",
            "2020-01-26 08:02:14,108 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:02:14,108 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:02:14,109 \tHypothesis: ( 1 : 1 ) Bhi nin mhan ha ha ha ha ha ha ha ha ha ha ha ha ha re egbe .\n",
            "2020-01-26 08:02:14,109 Example #3\n",
            "2020-01-26 08:02:14,109 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:02:14,109 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:02:14,110 \tHypothesis: ( 1 : 1 ) Be ha ha ha rẹkpa mhan rẹ ha rẹkpa mhan rẹ ha rẹkpa mhan .\n",
            "2020-01-26 08:02:14,110 Validation result (greedy) at epoch  12, step      500: bleu:   0.76, loss: 86351.7031, ppl:  45.9621, duration: 47.2545s\n",
            "2020-01-26 08:02:20,383 Epoch  12: total training loss 168.13\n",
            "2020-01-26 08:02:20,383 EPOCH 13\n",
            "2020-01-26 08:02:27,709 Epoch  13: total training loss 164.23\n",
            "2020-01-26 08:02:27,710 EPOCH 14\n",
            "2020-01-26 08:02:30,008 Epoch  14 Step:      600 Batch Loss:     3.553907 Tokens per Sec:    11914, Lr: 0.000300\n",
            "2020-01-26 08:03:16,042 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:03:16,043 Saving new checkpoint.\n",
            "2020-01-26 08:03:17,215 Example #0\n",
            "2020-01-26 08:03:17,216 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:03:17,216 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:03:17,216 \tHypothesis: Bhi ọsi ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne oga .\n",
            "2020-01-26 08:03:17,216 Example #1\n",
            "2020-01-26 08:03:17,216 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:03:17,216 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:03:17,217 \tHypothesis: Mhan da ha ha ha ha ha ha re emhin nin mhan rẹ ha ha ha re egbe .\n",
            "2020-01-26 08:03:17,217 Example #2\n",
            "2020-01-26 08:03:17,217 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:03:17,217 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:03:17,217 \tHypothesis: Bhi ọsi ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne\n",
            "2020-01-26 08:03:17,217 Example #3\n",
            "2020-01-26 08:03:17,217 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:03:17,217 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:03:17,218 \tHypothesis: Be bhọ mhan ha ha sabọ sabọ sabọ sabọ sabọ ha lu egbe ?\n",
            "2020-01-26 08:03:17,218 Validation result (greedy) at epoch  14, step      600: bleu:   0.55, loss: 83069.7344, ppl:  39.7390, duration: 47.2095s\n",
            "2020-01-26 08:03:22,015 Epoch  14: total training loss 160.68\n",
            "2020-01-26 08:03:22,016 EPOCH 15\n",
            "2020-01-26 08:03:29,243 Epoch  15: total training loss 156.83\n",
            "2020-01-26 08:03:29,244 EPOCH 16\n",
            "2020-01-26 08:03:33,296 Epoch  16 Step:      700 Batch Loss:     3.292420 Tokens per Sec:    12327, Lr: 0.000300\n",
            "2020-01-26 08:04:19,411 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:04:19,411 Saving new checkpoint.\n",
            "2020-01-26 08:04:20,600 Example #0\n",
            "2020-01-26 08:04:20,600 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:04:20,600 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:04:20,600 \tHypothesis: Ọle da wo ha lu .\n",
            "2020-01-26 08:04:20,601 Example #1\n",
            "2020-01-26 08:04:20,601 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:04:20,601 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:04:20,601 \tHypothesis: Mhan da ha rẹ ha rẹ ha rẹ ha rẹ ha rẹ ha rẹ ha lu bhi egbe .\n",
            "2020-01-26 08:04:20,601 Example #2\n",
            "2020-01-26 08:04:20,601 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:04:20,602 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:04:20,602 \tHypothesis: Bhi ọsi ijiẹmhin , ọle da ha rẹ ha yi ene biẹ mhan rẹ ha re obọ bhi ọne ọne agbọn nan , ọle da ha re obọ bhi ọne agbọn nan , ọle da ha re obọ bhi ọne ọne agbọn nan .\n",
            "2020-01-26 08:04:20,602 Example #3\n",
            "2020-01-26 08:04:20,602 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:04:20,602 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:04:20,602 \tHypothesis: Be bhọ mhan rẹ sabọ sabọ sabọ sabọ sabọ sabọ sabọ ha lu emhin nin mhan rẹ ha lu .\n",
            "2020-01-26 08:04:20,602 Validation result (greedy) at epoch  16, step      700: bleu:   1.23, loss: 79955.3750, ppl:  34.6147, duration: 47.3065s\n",
            "2020-01-26 08:04:23,769 Epoch  16: total training loss 153.85\n",
            "2020-01-26 08:04:23,769 EPOCH 17\n",
            "2020-01-26 08:04:31,068 Epoch  17: total training loss 150.37\n",
            "2020-01-26 08:04:31,068 EPOCH 18\n",
            "2020-01-26 08:04:36,604 Epoch  18 Step:      800 Batch Loss:     3.116080 Tokens per Sec:    12800, Lr: 0.000300\n",
            "2020-01-26 08:05:22,668 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:05:22,668 Saving new checkpoint.\n",
            "2020-01-26 08:05:23,789 Example #0\n",
            "2020-01-26 08:05:23,790 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:05:23,790 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:05:23,790 \tHypothesis: Ọle da wo ha yi mẹn .\n",
            "2020-01-26 08:05:23,790 Example #1\n",
            "2020-01-26 08:05:23,791 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:05:23,791 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:05:23,791 \tHypothesis: Mhan da ha rẹ ha re egbe nin ele rẹ ha lu emhin nin ele rẹ ha lu .\n",
            "2020-01-26 08:05:23,791 Example #2\n",
            "2020-01-26 08:05:23,791 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:05:23,791 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:05:23,791 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da yọle : “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ “ Ẹghe nin mhan rẹ ha re re obọ nin mhan rẹ ha re obọ , ọle rẹ ha re obọ\n",
            "2020-01-26 08:05:23,792 Example #3\n",
            "2020-01-26 08:05:23,792 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:05:23,792 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:05:23,792 \tHypothesis: Be bhọ mhan ha rẹ ha rẹkpa mhan rẹ ha lu emhin nin mhan rẹ ha lu .\n",
            "2020-01-26 08:05:23,792 Validation result (greedy) at epoch  18, step      800: bleu:   2.14, loss: 78357.0234, ppl:  32.2470, duration: 47.1879s\n",
            "2020-01-26 08:05:25,377 Epoch  18: total training loss 148.45\n",
            "2020-01-26 08:05:25,378 EPOCH 19\n",
            "2020-01-26 08:05:32,521 Epoch  19: total training loss 145.88\n",
            "2020-01-26 08:05:32,521 EPOCH 20\n",
            "2020-01-26 08:05:39,749 Epoch  20 Step:      900 Batch Loss:     2.919087 Tokens per Sec:    12611, Lr: 0.000300\n",
            "2020-01-26 08:06:25,750 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:06:25,751 Saving new checkpoint.\n",
            "2020-01-26 08:06:26,905 Example #0\n",
            "2020-01-26 08:06:26,906 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:06:26,906 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:06:26,906 \tHypothesis: Ọle da wo ha yi mẹn .\n",
            "2020-01-26 08:06:26,906 Example #1\n",
            "2020-01-26 08:06:26,907 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:06:26,907 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:06:26,907 \tHypothesis: Bhi ọsi ijiẹmhin , ene biẹ ọmọn nan rẹ ha mhọn ọne agbọn nan .\n",
            "2020-01-26 08:06:26,907 Example #2\n",
            "2020-01-26 08:06:26,907 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:06:26,907 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:06:26,907 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da ha yi eria ne ribhi ọne agbọn nan , ọle da ha yi eria ne bunbun .\n",
            "2020-01-26 08:06:26,907 Example #3\n",
            "2020-01-26 08:06:26,908 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:06:26,908 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:06:26,908 \tHypothesis: Mhan dẹ sabọ ha rẹ ha rẹ ha mhọn emhin nin mhan rẹ ha mhọn emhin ne ribhi oga .\n",
            "2020-01-26 08:06:26,908 Validation result (greedy) at epoch  20, step      900: bleu:   2.67, loss: 76491.7969, ppl:  29.6880, duration: 47.1589s\n",
            "2020-01-26 08:06:26,913 Epoch  20: total training loss 143.58\n",
            "2020-01-26 08:06:26,913 EPOCH 21\n",
            "2020-01-26 08:06:34,001 Epoch  21: total training loss 142.98\n",
            "2020-01-26 08:06:34,002 EPOCH 22\n",
            "2020-01-26 08:06:41,290 Epoch  22: total training loss 138.97\n",
            "2020-01-26 08:06:41,290 EPOCH 23\n",
            "2020-01-26 08:06:42,875 Epoch  23 Step:     1000 Batch Loss:     2.856789 Tokens per Sec:    12842, Lr: 0.000300\n",
            "2020-01-26 08:07:29,024 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:07:29,024 Saving new checkpoint.\n",
            "2020-01-26 08:07:30,244 Example #0\n",
            "2020-01-26 08:07:30,245 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:07:30,245 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:07:30,245 \tHypothesis: Ọle da wo ha yi mẹn bhi ọne otọ nan .\n",
            "2020-01-26 08:07:30,245 Example #1\n",
            "2020-01-26 08:07:30,246 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:07:30,246 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:07:30,246 \tHypothesis: Mhan dẹ sabọ ha re obọ rẹkhan emhin nin ele rẹ ha mhọn emhin ne bunbun , ele ki sabọ ha mhọn obọ bhi oga .\n",
            "2020-01-26 08:07:30,246 Example #2\n",
            "2020-01-26 08:07:30,247 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:07:30,247 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:07:30,247 \tHypothesis: ( John 4 : 1 ) Mhan da ha yi eria ne bunbun , ọle ki ha yi eria ne bunbun , ọle ki ha yi eria ne bunbun bhi ọne agbọn nan , ọle ki ha yi eria ne bunbun .\n",
            "2020-01-26 08:07:30,247 Example #3\n",
            "2020-01-26 08:07:30,248 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:07:30,248 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:07:30,248 \tHypothesis: Be bhọ mhan ha rẹ sabọ ha mhọn emhin nọn khẹke nin mhan rẹ ha mhọn emhin nin mhan rẹ ha lu ?\n",
            "2020-01-26 08:07:30,249 Validation result (greedy) at epoch  23, step     1000: bleu:   2.42, loss: 75343.2969, ppl:  28.2144, duration: 47.3732s\n",
            "2020-01-26 08:07:35,735 Epoch  23: total training loss 136.94\n",
            "2020-01-26 08:07:35,735 EPOCH 24\n",
            "2020-01-26 08:07:43,066 Epoch  24: total training loss 135.12\n",
            "2020-01-26 08:07:43,067 EPOCH 25\n",
            "2020-01-26 08:07:46,254 Epoch  25 Step:     1100 Batch Loss:     2.725771 Tokens per Sec:    12922, Lr: 0.000300\n",
            "2020-01-26 08:08:32,261 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:08:32,262 Saving new checkpoint.\n",
            "2020-01-26 08:08:33,379 Example #0\n",
            "2020-01-26 08:08:33,380 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:08:33,380 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:08:33,380 \tHypothesis: Ọle da wo ha yi mẹn .\n",
            "2020-01-26 08:08:33,380 Example #1\n",
            "2020-01-26 08:08:33,380 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:08:33,380 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:08:33,380 \tHypothesis: Ọ dẹ sabọ ha re egbe khọkhọ eria ne bunbun , ele ki sabọ ha re egbe khọkhọ eria ne bunbun .\n",
            "2020-01-26 08:08:33,381 Example #2\n",
            "2020-01-26 08:08:33,381 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:08:33,381 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:08:33,381 \tHypothesis: Mhan da ha yi eria ne bunbun , mhan ki ha re obọ kpa bhi ọne agbọn nan , ọle da ha yi eria ne bunbun .\n",
            "2020-01-26 08:08:33,381 Example #3\n",
            "2020-01-26 08:08:33,381 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:08:33,381 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:08:33,381 \tHypothesis: Mhan ha sabọ rẹkpa mhan rẹ sabọ ha lu emhin nin mhan ha lu .\n",
            "2020-01-26 08:08:33,381 Validation result (greedy) at epoch  25, step     1100: bleu:   2.75, loss: 74969.6875, ppl:  27.7510, duration: 47.1269s\n",
            "2020-01-26 08:08:37,239 Epoch  25: total training loss 129.85\n",
            "2020-01-26 08:08:37,240 EPOCH 26\n",
            "2020-01-26 08:08:44,469 Epoch  26: total training loss 131.09\n",
            "2020-01-26 08:08:44,469 EPOCH 27\n",
            "2020-01-26 08:08:49,356 Epoch  27 Step:     1200 Batch Loss:     2.751511 Tokens per Sec:    12659, Lr: 0.000300\n",
            "2020-01-26 08:09:35,517 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:09:35,517 Saving new checkpoint.\n",
            "2020-01-26 08:09:36,808 Example #0\n",
            "2020-01-26 08:09:36,809 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:09:36,809 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:09:36,809 \tHypothesis: Ọle da wo ha yi ọria nọn ha lu iwẹnna ọkanẹfan .\n",
            "2020-01-26 08:09:36,809 Example #1\n",
            "2020-01-26 08:09:36,809 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:09:36,809 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:09:36,810 \tHypothesis: Mhan ha sabọ miẹn ghe , uwẹ dẹ sabọ ha re egbe khọkhọ eria ne bunbun .\n",
            "2020-01-26 08:09:36,810 Example #2\n",
            "2020-01-26 08:09:36,810 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:09:36,810 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:09:36,810 \tHypothesis: Mhan da yọle : “ Mhan dẹ ha re obọ rẹkhan uhi nọn ribhi Baibo , mhan ki ha yi ẹbho ne bunbun , mhan ki ha re obọ rẹkhan ọlẹn . ” ” ” ” ( Matt .\n",
            "2020-01-26 08:09:36,811 Example #3\n",
            "2020-01-26 08:09:36,811 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:09:36,811 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:09:36,811 \tHypothesis: Mhan ha sabọ miẹn emhin ne bunbun bhi egbe .\n",
            "2020-01-26 08:09:36,811 Validation result (greedy) at epoch  27, step     1200: bleu:   3.16, loss: 73839.7266, ppl:  26.3952, duration: 47.4554s\n",
            "2020-01-26 08:09:39,028 Epoch  27: total training loss 129.69\n",
            "2020-01-26 08:09:39,028 EPOCH 28\n",
            "2020-01-26 08:09:46,304 Epoch  28: total training loss 127.06\n",
            "2020-01-26 08:09:46,304 EPOCH 29\n",
            "2020-01-26 08:09:52,660 Epoch  29 Step:     1300 Batch Loss:     2.944221 Tokens per Sec:    12920, Lr: 0.000300\n",
            "2020-01-26 08:10:38,727 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:10:38,727 Saving new checkpoint.\n",
            "2020-01-26 08:10:39,878 Example #0\n",
            "2020-01-26 08:10:39,879 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:10:39,879 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:10:39,879 \tHypothesis: Ọle da ha mhọn ọne iwẹnna nan .\n",
            "2020-01-26 08:10:39,879 Example #1\n",
            "2020-01-26 08:10:39,879 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:10:39,880 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:10:39,880 \tHypothesis: Bhiriọ , uwẹ dẹ sabọ rẹkpa uwẹ rẹ ha mhọn emhin nin uwẹ rẹ ha mhọn nọnsẹn .\n",
            "2020-01-26 08:10:39,880 Example #2\n",
            "2020-01-26 08:10:39,880 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:10:39,880 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:10:39,880 \tHypothesis: Mhan da ha re obọ rẹkhan uhi nọn ribhi Baibo , mhan ki ha re obọ rẹkhan adia nesi Jehova , bi ene ga Osẹnobulua .\n",
            "2020-01-26 08:10:39,880 Example #3\n",
            "2020-01-26 08:10:39,881 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:10:39,881 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:10:39,881 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ sabọ ha mhọn emhin nin mhan ha mhọn nọnsẹn .\n",
            "2020-01-26 08:10:39,881 Validation result (greedy) at epoch  29, step     1300: bleu:   3.21, loss: 73582.2969, ppl:  26.0957, duration: 47.2207s\n",
            "2020-01-26 08:10:40,551 Epoch  29: total training loss 125.88\n",
            "2020-01-26 08:10:40,551 EPOCH 30\n",
            "2020-01-26 08:10:47,611 Epoch  30: total training loss 123.65\n",
            "2020-01-26 08:10:47,611 EPOCH 31\n",
            "2020-01-26 08:10:54,774 Epoch  31: total training loss 122.03\n",
            "2020-01-26 08:10:54,775 EPOCH 32\n",
            "2020-01-26 08:10:55,747 Epoch  32 Step:     1400 Batch Loss:     2.719724 Tokens per Sec:    12781, Lr: 0.000300\n",
            "2020-01-26 08:11:41,993 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 08:11:41,994 Saving new checkpoint.\n",
            "2020-01-26 08:11:43,161 Example #0\n",
            "2020-01-26 08:11:43,162 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:11:43,162 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:11:43,162 \tHypothesis: Ọle da ha yi ikpe eva nin mẹn ha lu iwẹnna ọkanẹfan .\n",
            "2020-01-26 08:11:43,162 Example #1\n",
            "2020-01-26 08:11:43,162 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:11:43,162 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:11:43,162 \tHypothesis: Ene biẹ ọmọn nan wo ha mhọn ẹmhọn Agbejele nọnsi Jehova , ele da sabọ ha mhọn isẹhoa nin ele rẹ ha mhọn nọnsẹn .\n",
            "2020-01-26 08:11:43,163 Example #2\n",
            "2020-01-26 08:11:43,163 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:11:43,163 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:11:43,163 \tHypothesis: Mhan da ha re obọ rẹkhan uhi nọn ribhi Baibo yọle : “ Bha ji mhan rẹ ha re obọ rẹkhan adia nesi Jehova , ranmhude mhan ki ha re obọ rẹkhan ọlẹn , ranmhude mhan ki ha re obọ rẹkhan ọlẹn . ”\n",
            "2020-01-26 08:11:43,163 Example #3\n",
            "2020-01-26 08:11:43,163 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:11:43,163 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:11:43,163 \tHypothesis: Mhan dẹ sabọ ha re egbe khọkhọ eria ne bunbun .\n",
            "2020-01-26 08:11:43,164 Validation result (greedy) at epoch  32, step     1400: bleu:   3.39, loss: 72614.8125, ppl:  25.0002, duration: 47.4159s\n",
            "2020-01-26 08:11:49,257 Epoch  32: total training loss 120.11\n",
            "2020-01-26 08:11:49,257 EPOCH 33\n",
            "2020-01-26 08:11:56,537 Epoch  33: total training loss 118.35\n",
            "2020-01-26 08:11:56,537 EPOCH 34\n",
            "2020-01-26 08:11:59,151 Epoch  34 Step:     1500 Batch Loss:     2.812595 Tokens per Sec:    12506, Lr: 0.000300\n",
            "2020-01-26 08:12:45,318 Example #0\n",
            "2020-01-26 08:12:45,319 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:12:45,319 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:12:45,319 \tHypothesis: Mẹn da wo ha mhọn ẹmhọn oga .\n",
            "2020-01-26 08:12:45,320 Example #1\n",
            "2020-01-26 08:12:45,320 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:12:45,320 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:12:45,320 \tHypothesis: Ọnan wo manman re egbe khọkhọ iJehova bhi ẹghe nin uwẹ rẹ ha mhọn isẹhoa nin ọle rẹ ha mhọn idegbere .\n",
            "2020-01-26 08:12:45,321 Example #2\n",
            "2020-01-26 08:12:45,321 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:12:45,321 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:12:45,321 \tHypothesis: Mhan da yọle : “ Mhan dẹ ha re obọ rẹkhan uhi nọn ribhi Baibo , mhan ki ha re obọ rẹkhan ọlẹn , mhan ki ha re obọ rẹkhan ọlẹn , ranmhude mhan da yọle : “ Bha ha re obọ rẹkhan emhin nin ele . ”\n",
            "2020-01-26 08:12:45,321 Example #3\n",
            "2020-01-26 08:12:45,322 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:12:45,322 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:12:45,322 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ sabọ ha re egbe khọkhọ eria ne bunbun .\n",
            "2020-01-26 08:12:45,322 Validation result (greedy) at epoch  34, step     1500: bleu:   3.91, loss: 72724.7578, ppl:  25.1224, duration: 46.1704s\n",
            "2020-01-26 08:12:49,834 Epoch  34: total training loss 116.38\n",
            "2020-01-26 08:12:49,834 EPOCH 35\n",
            "2020-01-26 08:12:56,844 Epoch  35: total training loss 112.00\n",
            "2020-01-26 08:12:56,844 EPOCH 36\n",
            "2020-01-26 08:13:01,089 Epoch  36 Step:     1600 Batch Loss:     2.552213 Tokens per Sec:    12536, Lr: 0.000300\n",
            "2020-01-26 08:13:47,232 Example #0\n",
            "2020-01-26 08:13:47,233 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:13:47,233 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:13:47,233 \tHypothesis: Mẹn da wo ha mhọn ọne iwẹnna nan .\n",
            "2020-01-26 08:13:47,233 Example #1\n",
            "2020-01-26 08:13:47,234 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:13:47,234 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:13:47,234 \tHypothesis: Ọnan wo wo re ọne isẹhoa nan rẹ ha mhọn urẹọbhọ nọnsi Jehova .\n",
            "2020-01-26 08:13:47,234 Example #2\n",
            "2020-01-26 08:13:47,235 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:13:47,235 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:13:47,235 \tHypothesis: Mhan da ha yi ọkpa bhi ọne uhọnmhọn - ọta nan , mhan ki ha mhọn ọne ọta nan , mhan ki ha mhọn ọne otọ nan , ranmhude mhan da wo ha mhọn urẹọbhọ bi emhin ne bunbun .\n",
            "2020-01-26 08:13:47,235 Example #3\n",
            "2020-01-26 08:13:47,236 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:13:47,236 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:13:47,236 \tHypothesis: Mhan dẹ sabọ ha re egbe khọkhọ eria ne bunbun .\n",
            "2020-01-26 08:13:47,236 Validation result (greedy) at epoch  36, step     1600: bleu:   3.30, loss: 72694.6719, ppl:  25.0889, duration: 46.1465s\n",
            "2020-01-26 08:13:50,106 Epoch  36: total training loss 113.82\n",
            "2020-01-26 08:13:50,106 EPOCH 37\n",
            "2020-01-26 08:13:57,235 Epoch  37: total training loss 111.24\n",
            "2020-01-26 08:13:57,236 EPOCH 38\n",
            "2020-01-26 08:14:03,088 Epoch  38 Step:     1700 Batch Loss:     2.625222 Tokens per Sec:    12733, Lr: 0.000300\n",
            "2020-01-26 08:14:49,243 Example #0\n",
            "2020-01-26 08:14:49,245 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:14:49,245 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:14:49,245 \tHypothesis: Mẹn da ha yi ọkpa bhi ẹwẹ mẹn .\n",
            "2020-01-26 08:14:49,245 Example #1\n",
            "2020-01-26 08:14:49,245 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:14:49,246 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:14:49,246 \tHypothesis: Ene ikpẹ nan wo manman manman manman manman manman mun oga mhọn nọnsẹn .\n",
            "2020-01-26 08:14:49,246 Example #2\n",
            "2020-01-26 08:14:49,246 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:14:49,246 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:14:49,247 \tHypothesis: Mhan da ha yi ọkpa bhi Baibo yọle : “ Mhan dẹ ha mhọn urẹọbhọ bi ibhio mhan ne bunbun , ranmhude mhan khin , ranmhude mhan ki ha yi ọkpa bhi ọne otọ nan , ranmhude ọne agbọn nan ki vae . ”\n",
            "2020-01-26 08:14:49,247 Example #3\n",
            "2020-01-26 08:14:49,247 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:14:49,247 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:14:49,247 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn ọfure bi emhin nin mhan ha sabọ ha mhọnlẹn .\n",
            "2020-01-26 08:14:49,248 Validation result (greedy) at epoch  38, step     1700: bleu:   4.02, loss: 72884.2422, ppl:  25.3006, duration: 46.1595s\n",
            "2020-01-26 08:14:50,516 Epoch  38: total training loss 109.40\n",
            "2020-01-26 08:14:50,516 EPOCH 39\n",
            "2020-01-26 08:14:57,608 Epoch  39: total training loss 107.30\n",
            "2020-01-26 08:14:57,608 EPOCH 40\n",
            "2020-01-26 08:15:04,717 Epoch  40: total training loss 106.05\n",
            "2020-01-26 08:15:04,718 EPOCH 41\n",
            "2020-01-26 08:15:05,024 Epoch  41 Step:     1800 Batch Loss:     2.495653 Tokens per Sec:    12513, Lr: 0.000300\n",
            "2020-01-26 08:15:51,216 Example #0\n",
            "2020-01-26 08:15:51,217 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:15:51,217 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:15:51,217 \tHypothesis: Mẹn da wo ha mhọn urẹọbhọ bi mẹn .\n",
            "2020-01-26 08:15:51,217 Example #1\n",
            "2020-01-26 08:15:51,218 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:15:51,218 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:15:51,218 \tHypothesis: Ene ga iJehova wo re izebhudu nin ele , ele wo ha mhọn ọne isẹhoa nan rẹ ha mhọn ọne otọ nan .\n",
            "2020-01-26 08:15:51,218 Example #2\n",
            "2020-01-26 08:15:51,219 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:15:51,219 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:15:51,219 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da ha mhọn urẹọbhọ nọnsẹmhan , mhan ki dọ lẹn ghe mhan dẹ miẹn otọ , ranmhude mhan dẹ miẹn otọ , ranmhude mhan dẹ miẹn ọne isẹhoa nan rẹ re obọ rẹkhan ọlẹn . ”\n",
            "2020-01-26 08:15:51,219 Example #3\n",
            "2020-01-26 08:15:51,220 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:15:51,220 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:15:51,220 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn urẹọbhọ bi ọkhọnlẹn ọsaje .\n",
            "2020-01-26 08:15:51,220 Validation result (greedy) at epoch  41, step     1800: bleu:   4.19, loss: 73121.8594, ppl:  25.5685, duration: 46.1956s\n",
            "2020-01-26 08:15:58,006 Epoch  41: total training loss 103.96\n",
            "2020-01-26 08:15:58,006 EPOCH 42\n",
            "2020-01-26 08:16:05,126 Epoch  42: total training loss 102.08\n",
            "2020-01-26 08:16:05,126 EPOCH 43\n",
            "2020-01-26 08:16:07,006 Epoch  43 Step:     1900 Batch Loss:     1.994727 Tokens per Sec:    12047, Lr: 0.000300\n",
            "2020-01-26 08:16:53,220 Example #0\n",
            "2020-01-26 08:16:53,221 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:16:53,221 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:16:53,221 \tHypothesis: Mẹn da wo ha mhọn urẹọbhọ bi ọdọ mẹn .\n",
            "2020-01-26 08:16:53,222 Example #1\n",
            "2020-01-26 08:16:53,222 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:16:53,222 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:16:53,222 \tHypothesis: Ene ikpẹ nan wo manman manman re egbe khọkhọ iJehova bhi uwedẹ nin ele rẹ ha mhọn ọne isẹhoa nan rẹ rẹkpa ẹbho ne bunbun .\n",
            "2020-01-26 08:16:53,223 Example #2\n",
            "2020-01-26 08:16:53,223 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:16:53,223 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:16:53,223 \tHypothesis: Mhan da yọle : “ Mhan dẹ ha mhọn urẹọbhọ bi ọdọ , mhan da dọ ha mhọn urẹọbhọ da ẹbho nin mhan da ha mhọn da ẹbho , ranmhude mhan da yẹ ha mhọn urẹọbhọ bi azagba - uwa nọnsọle . ” ( Matt .\n",
            "2020-01-26 08:16:53,223 Example #3\n",
            "2020-01-26 08:16:53,224 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:16:53,224 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:16:53,224 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn isẹhoa nin mhan mhọnlẹn .\n",
            "2020-01-26 08:16:53,224 Validation result (greedy) at epoch  43, step     1900: bleu:   4.42, loss: 73039.2188, ppl:  25.4750, duration: 46.2183s\n",
            "2020-01-26 08:16:58,519 Epoch  43: total training loss 100.58\n",
            "2020-01-26 08:16:58,520 EPOCH 44\n",
            "2020-01-26 08:17:05,657 Epoch  44: total training loss 98.74\n",
            "2020-01-26 08:17:05,657 EPOCH 45\n",
            "2020-01-26 08:17:09,150 Epoch  45 Step:     2000 Batch Loss:     2.053864 Tokens per Sec:    12719, Lr: 0.000300\n",
            "2020-01-26 08:17:55,279 Example #0\n",
            "2020-01-26 08:17:55,280 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:17:55,280 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:17:55,280 \tHypothesis: Mẹn da wo ha mhọn ọne ẹmhọn nan .\n",
            "2020-01-26 08:17:55,281 Example #1\n",
            "2020-01-26 08:17:55,281 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:17:55,281 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:17:55,281 \tHypothesis: Emhin nọn mun oga mhọn nọnsẹn wo ha mhọn nọnsẹn rẹ ha mhọn urẹọbhọ bi emhin esili ne bunbun .\n",
            "2020-01-26 08:17:55,281 Example #2\n",
            "2020-01-26 08:17:55,282 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:17:55,282 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:17:55,282 \tHypothesis: Mhan da ha yi ọkpa bhi Baibo yọle : “ Mhan dẹ ha re obọ rẹkhan uhi bi emhin ne bunbun , ranmhude mhan mhọn urẹọbhọ bi azagba - uwa nọnsọle , ranmhude emhin nin ọle rẹ miẹn ọne otọ nan rebhe . ”\n",
            "2020-01-26 08:17:55,282 Example #3\n",
            "2020-01-26 08:17:55,283 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:17:55,283 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:17:55,283 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn ọfure ọsaje .\n",
            "2020-01-26 08:17:55,283 Validation result (greedy) at epoch  45, step     2000: bleu:   4.27, loss: 74022.8438, ppl:  26.6103, duration: 46.1327s\n",
            "2020-01-26 08:17:58,903 Epoch  45: total training loss 96.71\n",
            "2020-01-26 08:17:58,904 EPOCH 46\n",
            "2020-01-26 08:18:05,952 Epoch  46: total training loss 94.58\n",
            "2020-01-26 08:18:05,952 EPOCH 47\n",
            "2020-01-26 08:18:11,035 Epoch  47 Step:     2100 Batch Loss:     2.177901 Tokens per Sec:    13025, Lr: 0.000210\n",
            "2020-01-26 08:18:57,192 Example #0\n",
            "2020-01-26 08:18:57,193 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:18:57,193 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:18:57,194 \tHypothesis: Mẹn da wo ha mhọn ẹmhọn mẹn .\n",
            "2020-01-26 08:18:57,194 Example #1\n",
            "2020-01-26 08:18:57,194 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:18:57,194 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:18:57,194 \tHypothesis: Ene ikpẹ nan wo manman re egbe khọkhọ iJehova bhi iẹnlẹn nọnsele , ọle hi ene ga iJehova wo manman mun oga mhọn nọnsẹn . Ọle da wo ha mhọn ọne ahu nan .\n",
            "2020-01-26 08:18:57,195 Example #2\n",
            "2020-01-26 08:18:57,195 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:18:57,195 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:18:57,195 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan dẹ sabọ re obọ rẹkhan ọlẹn , mhan ki dọ ha mhọn ọne isẹhoa nan rẹ sẹbhi eji ọle da yọle : “ Bha ha yi ikpe ne bunbun . ”\n",
            "2020-01-26 08:18:57,196 Example #3\n",
            "2020-01-26 08:18:57,196 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:18:57,196 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:18:57,196 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn ọfure ọsaje nin mhan mhọnlẹn .\n",
            "2020-01-26 08:18:57,196 Validation result (greedy) at epoch  47, step     2100: bleu:   4.64, loss: 74151.1016, ppl:  26.7621, duration: 46.1614s\n",
            "2020-01-26 08:18:59,148 Epoch  47: total training loss 90.88\n",
            "2020-01-26 08:18:59,148 EPOCH 48\n",
            "2020-01-26 08:19:06,191 Epoch  48: total training loss 89.59\n",
            "2020-01-26 08:19:06,192 EPOCH 49\n",
            "2020-01-26 08:19:13,173 Epoch  49 Step:     2200 Batch Loss:     2.089329 Tokens per Sec:    12720, Lr: 0.000210\n",
            "2020-01-26 08:19:59,355 Example #0\n",
            "2020-01-26 08:19:59,357 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:19:59,357 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:19:59,357 \tHypothesis: Mẹn da wo ha mhọn ẹmhọn mẹn .\n",
            "2020-01-26 08:19:59,357 Example #1\n",
            "2020-01-26 08:19:59,358 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:19:59,358 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:19:59,358 \tHypothesis: Ene ikpẹ nan wo re izebhudu nin ele , ele wo manman mun udu nyan iJehova bhi iẹnlẹn nọnsele . Ọle da wo ha mhọn idegbere .\n",
            "2020-01-26 08:19:59,358 Example #2\n",
            "2020-01-26 08:19:59,359 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:19:59,359 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:19:59,359 \tHypothesis: Ẹlẹnan nian , mhan ki dọ kere ghe , Jehova mhọn urẹọbhọ bi emhin nin ọle rẹ ha ribhi ẹkẹ agbotu nọnsọle , ọle ki dọ ha mhọn ọlẹn bhi otọ ghe , ranmhude mhan da yẹ re ọne otọ nan rebhe . ” ( Matt .\n",
            "2020-01-26 08:19:59,359 Example #3\n",
            "2020-01-26 08:19:59,360 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:19:59,360 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:19:59,360 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn urẹọbhọ bi ọkhọnlẹn ọsaje .\n",
            "2020-01-26 08:19:59,360 Validation result (greedy) at epoch  49, step     2200: bleu:   4.81, loss: 74790.0938, ppl:  27.5309, duration: 46.1865s\n",
            "2020-01-26 08:19:59,528 Epoch  49: total training loss 90.24\n",
            "2020-01-26 08:19:59,528 EPOCH 50\n",
            "2020-01-26 08:20:06,616 Epoch  50: total training loss 89.23\n",
            "2020-01-26 08:20:06,616 EPOCH 51\n",
            "2020-01-26 08:20:13,733 Epoch  51: total training loss 87.68\n",
            "2020-01-26 08:20:13,733 EPOCH 52\n",
            "2020-01-26 08:20:15,193 Epoch  52 Step:     2300 Batch Loss:     1.752498 Tokens per Sec:    13183, Lr: 0.000210\n",
            "2020-01-26 08:21:01,361 Example #0\n",
            "2020-01-26 08:21:01,362 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:21:01,362 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:21:01,362 \tHypothesis: Mẹn da ha mhọn urẹọbhọ bi mẹn .\n",
            "2020-01-26 08:21:01,362 Example #1\n",
            "2020-01-26 08:21:01,363 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:21:01,363 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:21:01,363 \tHypothesis: Ene ikpẹ nan wo manman re izebhudu nin ele , ele wo manman mun oga mhọn nọnsẹn rẹ ha mhọn nọnsẹn . Ọle da ene ẹbho nan erọnmhọn ji Jehova .\n",
            "2020-01-26 08:21:01,363 Example #2\n",
            "2020-01-26 08:21:01,364 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:21:01,364 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:21:01,364 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan ki dọ lẹn otọ emhin rebhe ne ribhi Baibo , ranmhude mhan guanọ nin ọle dọ ha yi ọkpa bhi ọne agbọn nan , ranmhude ọne agbọn nan da sẹyẹ wo ha yi ojie . ( Matt .\n",
            "2020-01-26 08:21:01,364 Example #3\n",
            "2020-01-26 08:21:01,365 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:21:01,365 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:21:01,365 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn ọfure ọsaje .\n",
            "2020-01-26 08:21:01,365 Validation result (greedy) at epoch  52, step     2300: bleu:   4.99, loss: 75371.0000, ppl:  28.2491, duration: 46.1714s\n",
            "2020-01-26 08:21:06,924 Epoch  52: total training loss 84.63\n",
            "2020-01-26 08:21:06,924 EPOCH 53\n",
            "2020-01-26 08:21:14,018 Epoch  53: total training loss 85.55\n",
            "2020-01-26 08:21:14,018 EPOCH 54\n",
            "2020-01-26 08:21:17,127 Epoch  54 Step:     2400 Batch Loss:     2.010780 Tokens per Sec:    13031, Lr: 0.000210\n",
            "2020-01-26 08:22:03,302 Example #0\n",
            "2020-01-26 08:22:03,303 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:22:03,303 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:22:03,303 \tHypothesis: Mẹn da ha mhọn emhanmhan nin mẹn ha mhọnlẹn .\n",
            "2020-01-26 08:22:03,303 Example #1\n",
            "2020-01-26 08:22:03,304 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:22:03,304 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:22:03,304 \tHypothesis: Ene ikpẹ nan wo manman re egbe khọkhọ iJehova bhi iẹnlẹn ọsi eria ne bunbun . Ọle wo manman ha mhọn urẹọbhọ bi ene ẹbho nan rebhe .\n",
            "2020-01-26 08:22:03,304 Example #2\n",
            "2020-01-26 08:22:03,305 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:22:03,305 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:22:03,305 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , Jehova da taman mhan ghe , nin mhan hẹi ha mhọn ọne isẹhoa nan rẹ lu emhin , ranmhude ọnan da yẹ re izebhudu nin ẹbho ne bunbun . ( Matt .\n",
            "2020-01-26 08:22:03,305 Example #3\n",
            "2020-01-26 08:22:03,306 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:22:03,306 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:22:03,306 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn okugbe nọn khiale .\n",
            "2020-01-26 08:22:03,306 Validation result (greedy) at epoch  54, step     2400: bleu:   5.14, loss: 76095.4609, ppl:  29.1710, duration: 46.1789s\n",
            "2020-01-26 08:22:07,274 Epoch  54: total training loss 84.40\n",
            "2020-01-26 08:22:07,275 EPOCH 55\n",
            "2020-01-26 08:22:14,369 Epoch  55: total training loss 83.18\n",
            "2020-01-26 08:22:14,369 EPOCH 56\n",
            "2020-01-26 08:22:19,048 Epoch  56 Step:     2500 Batch Loss:     1.566270 Tokens per Sec:    12842, Lr: 0.000210\n",
            "2020-01-26 08:23:05,241 Example #0\n",
            "2020-01-26 08:23:05,242 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:23:05,242 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:23:05,243 \tHypothesis: Mẹn da ha riale ghe mẹn dẹ mun mẹn ọbhi egbe .\n",
            "2020-01-26 08:23:05,243 Example #1\n",
            "2020-01-26 08:23:05,243 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:23:05,244 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:23:05,244 \tHypothesis: Ene ikpẹ nan wo manman rẹkpa ele rẹ lẹn ghe , ele wo mun oga mhọn nọnsẹn nin ele rẹ ha mhọn ọne ahu nan . Ọle wo manman kaka .\n",
            "2020-01-26 08:23:05,244 Example #2\n",
            "2020-01-26 08:23:05,244 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:23:05,244 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:23:05,245 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan dẹ sabọ re izebhudu nin ibhio mhan ne ribhi ọne agbọn nan , ranmhude mhan khin , ranmhude ọnan da yọle : “ Bha wo re ọne isẹhoa nan rẹ lu emhin ne imhẹn . ”\n",
            "2020-01-26 08:23:05,245 Example #3\n",
            "2020-01-26 08:23:05,245 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:23:05,245 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:23:05,245 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha lu emhin kpataki .\n",
            "2020-01-26 08:23:05,246 Validation result (greedy) at epoch  56, step     2500: bleu:   5.46, loss: 76564.3125, ppl:  29.7836, duration: 46.1973s\n",
            "2020-01-26 08:23:07,622 Epoch  56: total training loss 82.02\n",
            "2020-01-26 08:23:07,623 EPOCH 57\n",
            "2020-01-26 08:23:14,710 Epoch  57: total training loss 80.56\n",
            "2020-01-26 08:23:14,710 EPOCH 58\n",
            "2020-01-26 08:23:20,983 Epoch  58 Step:     2600 Batch Loss:     2.045136 Tokens per Sec:    12897, Lr: 0.000210\n",
            "2020-01-26 08:24:07,178 Example #0\n",
            "2020-01-26 08:24:07,178 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:24:07,179 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:24:07,179 \tHypothesis: Mẹn da wo ha lu iwẹnna ọkanẹfan .\n",
            "2020-01-26 08:24:07,179 Example #1\n",
            "2020-01-26 08:24:07,179 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:24:07,180 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:24:07,180 \tHypothesis: Ene ikpẹ nan wo manman re izebhudu nin ele , ọle ene ga iJehova wo manman ha mhọn urẹọbhọ da ẹbho ne bunbun .\n",
            "2020-01-26 08:24:07,180 Example #2\n",
            "2020-01-26 08:24:07,180 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:24:07,181 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:24:07,181 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan da ha mhọn oyẹẹ da ẹbho ne ribhi ẹkẹ agbotu nọnsọle , ranmhude mhan da ha yi ọmọẹ ọle . ( Matt .\n",
            "2020-01-26 08:24:07,181 Example #3\n",
            "2020-01-26 08:24:07,182 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:24:07,182 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:24:07,182 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha lu emhin kpataki .\n",
            "2020-01-26 08:24:07,182 Validation result (greedy) at epoch  58, step     2600: bleu:   5.29, loss: 77054.5312, ppl:  30.4379, duration: 46.1992s\n",
            "2020-01-26 08:24:07,996 Epoch  58: total training loss 79.56\n",
            "2020-01-26 08:24:07,996 EPOCH 59\n",
            "2020-01-26 08:24:15,070 Epoch  59: total training loss 77.70\n",
            "2020-01-26 08:24:15,070 EPOCH 60\n",
            "2020-01-26 08:24:22,125 Epoch  60: total training loss 76.92\n",
            "2020-01-26 08:24:22,125 EPOCH 61\n",
            "2020-01-26 08:24:22,936 Epoch  61 Step:     2700 Batch Loss:     1.487003 Tokens per Sec:    11664, Lr: 0.000147\n",
            "2020-01-26 08:25:09,161 Example #0\n",
            "2020-01-26 08:25:09,162 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:25:09,162 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:25:09,163 \tHypothesis: Mẹn da wo ha mhọn emhanmhan nin mẹn ha mhọnlẹn .\n",
            "2020-01-26 08:25:09,163 Example #1\n",
            "2020-01-26 08:25:09,163 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:25:09,163 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:25:09,163 \tHypothesis: Ene ikpẹ nan wo manman re egbe khọkhọ iJehova bhi iẹnlẹn nọnsele , ọle hi ene ẹbho ne mhọn ọyẹẹ da ẹbho ne mhọn akan - ehọ .\n",
            "2020-01-26 08:25:09,164 Example #2\n",
            "2020-01-26 08:25:09,164 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:25:09,164 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:25:09,164 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan da ha mhọn oyẹẹ da ẹbho ne ribhi ẹkẹ agbotu , ranmhude mhan wo ha mhọn elin ọsi Agbejele , ranmhude emhin nin ọle daghe ghe ọle daghe emhin nọn imhẹn . ( Matt .\n",
            "2020-01-26 08:25:09,164 Example #3\n",
            "2020-01-26 08:25:09,165 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:25:09,165 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:25:09,165 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn okugbe nọn khua .\n",
            "2020-01-26 08:25:09,165 Validation result (greedy) at epoch  61, step     2700: bleu:   5.55, loss: 77714.9219, ppl:  31.3421, duration: 46.2290s\n",
            "2020-01-26 08:25:15,491 Epoch  61: total training loss 75.74\n",
            "2020-01-26 08:25:15,492 EPOCH 62\n",
            "2020-01-26 08:25:22,583 Epoch  62: total training loss 75.10\n",
            "2020-01-26 08:25:22,583 EPOCH 63\n",
            "2020-01-26 08:25:24,964 Epoch  63 Step:     2800 Batch Loss:     1.604341 Tokens per Sec:    12225, Lr: 0.000147\n",
            "2020-01-26 08:26:11,199 Example #0\n",
            "2020-01-26 08:26:11,200 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:26:11,200 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:26:11,200 \tHypothesis: Mẹn da wo ha ghọnghọn ranmhude mẹn guanọ nin mẹn rẹ lu .\n",
            "2020-01-26 08:26:11,201 Example #1\n",
            "2020-01-26 08:26:11,201 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:26:11,201 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:26:11,201 \tHypothesis: Ene ikpẹ nan wo manman rẹkpa ene ga iJehova bhi uwedẹ nin ele rẹ wo ha mhọn okpẹhio , ele wo wo mun ekpẹn nin ene ga iJehova .\n",
            "2020-01-26 08:26:11,201 Example #2\n",
            "2020-01-26 08:26:11,202 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:26:11,202 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:26:11,202 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan da ha mhọn oyẹẹ da ẹbho ne ribhi ẹkẹ agbotu bi emhin rebhe , nin ọle da sẹyẹ ha yi ọkpa bhi ọne agbọn nan . ( Matt .\n",
            "2020-01-26 08:26:11,202 Example #3\n",
            "2020-01-26 08:26:11,203 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:26:11,203 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:26:11,203 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn okugbe nọn khiale .\n",
            "2020-01-26 08:26:11,203 Validation result (greedy) at epoch  63, step     2800: bleu:   5.55, loss: 78130.3047, ppl:  31.9246, duration: 46.2389s\n",
            "2020-01-26 08:26:15,939 Epoch  63: total training loss 74.24\n",
            "2020-01-26 08:26:15,939 EPOCH 64\n",
            "2020-01-26 08:26:22,978 Epoch  64: total training loss 73.16\n",
            "2020-01-26 08:26:22,979 EPOCH 65\n",
            "2020-01-26 08:26:26,949 Epoch  65 Step:     2900 Batch Loss:     1.575985 Tokens per Sec:    12829, Lr: 0.000147\n",
            "2020-01-26 08:27:13,220 Example #0\n",
            "2020-01-26 08:27:13,221 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:27:13,222 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:27:13,222 \tHypothesis: Mẹn da wo ha ghọnghọn .\n",
            "2020-01-26 08:27:13,222 Example #1\n",
            "2020-01-26 08:27:13,223 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:27:13,223 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:27:13,223 \tHypothesis: Ene ikpẹ nan wo rẹkpa ene ga iJehova bhi iẹnlẹn nọnsele , ọle ene ga iJehova wo manman ha mhọn ahu nin ele rẹ wo ha mhọn okpẹhio .\n",
            "2020-01-26 08:27:13,223 Example #2\n",
            "2020-01-26 08:27:13,223 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:27:13,224 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:27:13,224 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , Jehova da re oyẹẹ man ẹbho , nin ele da sabọ ha yi ẹbho ne ribhi ọne agbọn nan , ranmhude ọnan da re izebhudu nin ele . ( Matt .\n",
            "2020-01-26 08:27:13,224 Example #3\n",
            "2020-01-26 08:27:13,224 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:27:13,225 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:27:13,225 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mun oga mhọn nọnsẹn .\n",
            "2020-01-26 08:27:13,225 Validation result (greedy) at epoch  65, step     2900: bleu:   5.43, loss: 78853.4531, ppl:  32.9645, duration: 46.2759s\n",
            "2020-01-26 08:27:16,394 Epoch  65: total training loss 72.62\n",
            "2020-01-26 08:27:16,394 EPOCH 66\n",
            "2020-01-26 08:27:23,459 Epoch  66: total training loss 71.77\n",
            "2020-01-26 08:27:23,460 EPOCH 67\n",
            "2020-01-26 08:27:28,981 Epoch  67 Step:     3000 Batch Loss:     1.416852 Tokens per Sec:    12840, Lr: 0.000147\n",
            "2020-01-26 08:28:15,253 Example #0\n",
            "2020-01-26 08:28:15,254 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:28:15,254 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:28:15,254 \tHypothesis: Mẹn da wo ha ghọnghọn .\n",
            "2020-01-26 08:28:15,254 Example #1\n",
            "2020-01-26 08:28:15,255 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:28:15,255 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:28:15,255 \tHypothesis: Ene ikpẹ nan wo rẹkpa ene biẹ ọmọn bi ibhio mhan nan wo lẹn ghe , ele wo hoẹmhọn iJehova . Ọle wo ha mhọn urẹọbhọ nin ele mhọnlẹn .\n",
            "2020-01-26 08:28:15,255 Example #2\n",
            "2020-01-26 08:28:15,256 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:28:15,256 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:28:15,256 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , Jehova wo re ọne isẹhoa nan rẹ lu iriọ yẹ , mhan ki dọ taman ele ghe , ọiyi ebe ebe rebhe nin ọle ha rẹ re obọ kpa bhi ọne agbọn nan . ” ( Matt .\n",
            "2020-01-26 08:28:15,256 Example #3\n",
            "2020-01-26 08:28:15,256 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:28:15,256 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:28:15,257 \tHypothesis: Emhin kpataki nọn nin mhan rẹ sabọ rẹkpa mhan rẹ ha mhọn okugbe .\n",
            "2020-01-26 08:28:15,257 Validation result (greedy) at epoch  67, step     3000: bleu:   5.81, loss: 79436.3047, ppl:  33.8273, duration: 46.2752s\n",
            "2020-01-26 08:28:16,796 Epoch  67: total training loss 70.84\n",
            "2020-01-26 08:28:16,796 EPOCH 68\n",
            "2020-01-26 08:28:23,845 Epoch  68: total training loss 70.11\n",
            "2020-01-26 08:28:23,846 EPOCH 69\n",
            "2020-01-26 08:28:30,885 Epoch  69 Step:     3100 Batch Loss:     1.464702 Tokens per Sec:    12948, Lr: 0.000147\n",
            "2020-01-26 08:29:17,118 Example #0\n",
            "2020-01-26 08:29:17,119 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:29:17,120 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:29:17,120 \tHypothesis: Mẹn da wo ha ghọnghọn .\n",
            "2020-01-26 08:29:17,120 Example #1\n",
            "2020-01-26 08:29:17,120 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:29:17,121 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:29:17,121 \tHypothesis: Ene ikpẹ nan wo rẹkpa ene ga iJehova bhi ẹghe nin ele rẹ wo ha mhọn okpẹhio , ele wo mun ekpẹn nin ẹbho ne bunbun .\n",
            "2020-01-26 08:29:17,121 Example #2\n",
            "2020-01-26 08:29:17,121 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:29:17,121 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:29:17,121 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn sẹbhọ ghe , Jehova da re ọne isẹhoa nan rẹ lu emhin rebhe , nin mhan rẹ re izebhudu nin ẹbho ne ribhi ọne agbọn nan , ọle da sabọ re ọne otọ nan rebhe . ”\n",
            "2020-01-26 08:29:17,122 Example #3\n",
            "2020-01-26 08:29:17,122 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:29:17,122 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:29:17,122 \tHypothesis: Ọkhẹke nin mhan dọnmhegbe rẹ ha tuẹn okugbe nọnsẹmhan okhun .\n",
            "2020-01-26 08:29:17,122 Validation result (greedy) at epoch  69, step     3100: bleu:   5.59, loss: 80199.3047, ppl:  34.9910, duration: 46.2372s\n",
            "2020-01-26 08:29:17,126 Epoch  69: total training loss 69.29\n",
            "2020-01-26 08:29:17,126 EPOCH 70\n",
            "2020-01-26 08:29:24,193 Epoch  70: total training loss 68.78\n",
            "2020-01-26 08:29:24,193 EPOCH 71\n",
            "2020-01-26 08:29:31,245 Epoch  71: total training loss 68.11\n",
            "2020-01-26 08:29:31,245 EPOCH 72\n",
            "2020-01-26 08:29:32,829 Epoch  72 Step:     3200 Batch Loss:     1.396934 Tokens per Sec:    12896, Lr: 0.000147\n",
            "2020-01-26 08:30:18,993 Example #0\n",
            "2020-01-26 08:30:18,994 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 08:30:18,994 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 08:30:18,994 \tHypothesis: Mẹn da yẹ ha ghọnghọn ranmhude mẹn ki munhẹn ha lu .\n",
            "2020-01-26 08:30:18,994 Example #1\n",
            "2020-01-26 08:30:18,994 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 08:30:18,995 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 08:30:18,995 \tHypothesis: Ene ikpẹ nan wo rẹkpa ene ga iJehova bhi ẹghe nin ele rẹ dọ lẹn ghe Osẹnobulua wo hoẹmhọn ẹbho nekẹle . Ọle wo ha mhọn eji ele rẹ ka ka tẹmhọnlẹn .\n",
            "2020-01-26 08:30:18,995 Example #2\n",
            "2020-01-26 08:30:18,995 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 08:30:18,996 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 08:30:18,996 \tHypothesis: Ẹlẹnan nian , mhan ki dọ lẹn ghe , mhan da ha re oyẹẹ man ẹbho , yẹ daghe ghe , nin ele rẹ ha ribhi ọne agbọn nan , ọle hi : “ ene ga ọle a da miẹn elele bhi ọne agbọn nan . ”\n",
            "2020-01-26 08:30:18,996 Example #3\n",
            "2020-01-26 08:30:18,996 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 08:30:18,996 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 08:30:18,997 \tHypothesis: Mhan dẹ yẹ sabọ rẹkpa mhan rẹ ha mun oga mhọn nọnsẹn .\n",
            "2020-01-26 08:30:18,997 Validation result (greedy) at epoch  72, step     3200: bleu:   5.48, loss: 80505.1406, ppl:  35.4686, duration: 46.1670s\n",
            "2020-01-26 08:30:24,488 Epoch  72: total training loss 67.09\n",
            "2020-01-26 08:30:24,489 EPOCH 73\n",
            "2020-01-26 08:30:31,521 Epoch  73: total training loss 66.03\n",
            "2020-01-26 08:30:31,522 EPOCH 74\n",
            "2020-01-26 08:30:34,707 Epoch  74 Step:     3300 Batch Loss:     1.702576 Tokens per Sec:    13220, Lr: 0.000103\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n",
            "    \"__main__\", mod_spec)\n",
            "  File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n",
            "    exec(code, run_globals)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/__main__.py\", line 41, in <module>\n",
            "    main()\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/__main__.py\", line 29, in main\n",
            "    train(cfg_file=args.config_path)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/training.py\", line 596, in train\n",
            "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/training.py\", line 338, in train_and_validate\n",
            "    batch_type=self.eval_batch_type\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/prediction.py\", line 106, in validate_on_data\n",
            "    max_output_length=max_output_length)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/model.py\", line 170, in run_batch\n",
            "    max_output_length=max_output_length)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/search.py\", line 43, in greedy\n",
            "    decoder, encoder_output, encoder_hidden)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/search.py\", line 139, in transformer_greedy\n",
            "    trg_mask=trg_mask\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\", line 541, in __call__\n",
            "    result = self.forward(*input, **kwargs)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/joeynmt/decoders.py\", line 514, in forward\n",
            "    trg_embed.size(1)).type_as(trg_mask)\n",
            "KeyboardInterrupt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "outputId": "a42675ce-50be-4ad1-ee11-877f6abfc194",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        }
      },
      "source": [
        "# Output our validation accuracy epoch 1-74\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 100\tLoss: 110740.54688\tPPL: 135.49428\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 200\tLoss: 103501.96875\tPPL: 98.30320\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 300\tLoss: 96958.10156\tPPL: 73.55095\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 400\tLoss: 90492.06250\tPPL: 55.22141\tbleu: 0.17124\tLR: 0.00030000\t*\n",
            "Steps: 500\tLoss: 86351.70312\tPPL: 45.96205\tbleu: 0.76432\tLR: 0.00030000\t*\n",
            "Steps: 600\tLoss: 83069.73438\tPPL: 39.73896\tbleu: 0.55221\tLR: 0.00030000\t*\n",
            "Steps: 700\tLoss: 79955.37500\tPPL: 34.61469\tbleu: 1.23362\tLR: 0.00030000\t*\n",
            "Steps: 800\tLoss: 78357.02344\tPPL: 32.24704\tbleu: 2.14242\tLR: 0.00030000\t*\n",
            "Steps: 900\tLoss: 76491.79688\tPPL: 29.68803\tbleu: 2.67077\tLR: 0.00030000\t*\n",
            "Steps: 1000\tLoss: 75343.29688\tPPL: 28.21441\tbleu: 2.41983\tLR: 0.00030000\t*\n",
            "Steps: 1100\tLoss: 74969.68750\tPPL: 27.75099\tbleu: 2.74535\tLR: 0.00030000\t*\n",
            "Steps: 1200\tLoss: 73839.72656\tPPL: 26.39521\tbleu: 3.15579\tLR: 0.00030000\t*\n",
            "Steps: 1300\tLoss: 73582.29688\tPPL: 26.09572\tbleu: 3.21157\tLR: 0.00030000\t*\n",
            "Steps: 1400\tLoss: 72614.81250\tPPL: 25.00021\tbleu: 3.39397\tLR: 0.00030000\t*\n",
            "Steps: 1500\tLoss: 72724.75781\tPPL: 25.12235\tbleu: 3.90548\tLR: 0.00030000\t\n",
            "Steps: 1600\tLoss: 72694.67188\tPPL: 25.08887\tbleu: 3.29865\tLR: 0.00030000\t\n",
            "Steps: 1700\tLoss: 72884.24219\tPPL: 25.30059\tbleu: 4.02374\tLR: 0.00030000\t\n",
            "Steps: 1800\tLoss: 73121.85938\tPPL: 25.56849\tbleu: 4.19265\tLR: 0.00030000\t\n",
            "Steps: 1900\tLoss: 73039.21875\tPPL: 25.47499\tbleu: 4.41915\tLR: 0.00030000\t\n",
            "Steps: 2000\tLoss: 74022.84375\tPPL: 26.61034\tbleu: 4.26710\tLR: 0.00021000\t\n",
            "Steps: 2100\tLoss: 74151.10156\tPPL: 26.76206\tbleu: 4.64173\tLR: 0.00021000\t\n",
            "Steps: 2200\tLoss: 74790.09375\tPPL: 27.53094\tbleu: 4.80778\tLR: 0.00021000\t\n",
            "Steps: 2300\tLoss: 75371.00000\tPPL: 28.24909\tbleu: 4.99349\tLR: 0.00021000\t\n",
            "Steps: 2400\tLoss: 76095.46094\tPPL: 29.17100\tbleu: 5.14330\tLR: 0.00021000\t\n",
            "Steps: 2500\tLoss: 76564.31250\tPPL: 29.78362\tbleu: 5.45634\tLR: 0.00021000\t\n",
            "Steps: 2600\tLoss: 77054.53125\tPPL: 30.43791\tbleu: 5.29034\tLR: 0.00014700\t\n",
            "Steps: 2700\tLoss: 77714.92188\tPPL: 31.34212\tbleu: 5.54971\tLR: 0.00014700\t\n",
            "Steps: 2800\tLoss: 78130.30469\tPPL: 31.92457\tbleu: 5.55277\tLR: 0.00014700\t\n",
            "Steps: 2900\tLoss: 78853.45312\tPPL: 32.96452\tbleu: 5.42751\tLR: 0.00014700\t\n",
            "Steps: 3000\tLoss: 79436.30469\tPPL: 33.82732\tbleu: 5.81458\tLR: 0.00014700\t\n",
            "Steps: 3100\tLoss: 80199.30469\tPPL: 34.99101\tbleu: 5.58647\tLR: 0.00014700\t\n",
            "Steps: 3200\tLoss: 80505.14062\tPPL: 35.46862\tbleu: 5.48380\tLR: 0.00010290\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "outputId": "e2a7d710-9045-4c5c-b5c9-3ae8a1b944bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-01-26 08:31:49,176 Hello! This is Joey-NMT.\n",
            "2020-01-26 08:32:09,525  dev bleu:   3.39 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-01-26 08:32:13,914 test bleu:   5.30 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}