{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oGRmDELn7Az0"
   },
   "outputs": [],
   "source": [
    "\"\"\"from google.colab import drive\n",
    "drive.mount('/content/drive')\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"tiv\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"jw300-baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "\n",
    "# This will save it to a folder in our gdrive instead!\n",
    "!mkdir -p \"en_tiv/$src-$tgt-$tag\"\n",
    "os.environ[\"experiment_path\"] = \"en_tiv/%s-%s-%s\" % (source_language, target_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kBSgJHEw7Nvx"
   },
   "outputs": [],
   "source": [
    "!mkdir -p \"en_tiv/$src-$tgt-$tag\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "en_tiv/en-tiv-jw300-baseline\r\n"
     ]
    }
   ],
   "source": [
    "!echo $experiment_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gA75Fs9ys8Y9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: opustools-pkg in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (0.0.52)\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install opus-tools\n",
    "! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xq-tDZVks7ZD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-tiv.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "   2 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-tiv.xml.gz\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "  25 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/tiv.zip\n",
      "\n",
      " 290 MB Total size\n",
      "./JW300_latest_xml_en-tiv.xml.gz ... 100% of 2 MB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "./JW300_latest_xml_tiv.zip ... 100% of 25 MB\n"
     ]
    }
   ],
   "source": [
    "# Downloading our corpus\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n48GDRnP8y2G"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-12 15:17:58--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.24.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.24.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en’\n",
      "\n",
      "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2020-02-12 15:17:58 (26.3 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
      "\n",
      "--2020-02-12 15:17:58--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-tiv.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.24.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.24.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 26584 (26K) [text/plain]\n",
      "Saving to: ‘test.en-tiv.en’\n",
      "\n",
      "test.en-tiv.en      100%[===================>]  25.96K  --.-KB/s    in 0.001s  \n",
      "\n",
      "2020-02-12 15:17:59 (22.7 MB/s) - ‘test.en-tiv.en’ saved [26584/26584]\n",
      "\n",
      "--2020-02-12 15:17:59--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-tiv.tiv\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.24.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.24.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 30763 (30K) [text/plain]\n",
      "Saving to: ‘test.en-tiv.tiv’\n",
      "\n",
      "test.en-tiv.tiv     100%[===================>]  30.04K  --.-KB/s    in 0.001s  \n",
      "\n",
      "2020-02-12 15:17:59 (24.6 MB/s) - ‘test.en-tiv.tiv’ saved [30763/30763]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NqDG-CI28y2L"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3CNdwLBCfSIl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 2535/209778 lines since contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Questions From Readers</td>\n",
       "      <td>Mbampin Mba Mbaôron Takerada Ne Ve Pin la</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How seriously should Christians view an engage...</td>\n",
       "      <td>Gba u ityendezwa i nomsoor vea kwase ve er u v...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>An engagement to marry is a cause for happines...</td>\n",
       "      <td>Ityendezwa i̱ nomsoor vea kwase ve er ér vea v...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence  \\\n",
       "0                             Questions From Readers   \n",
       "1  How seriously should Christians view an engage...   \n",
       "2  An engagement to marry is a cause for happines...   \n",
       "\n",
       "                                     target_sentence  \n",
       "0          Mbampin Mba Mbaôron Takerada Ne Ve Pin la  \n",
       "1  Gba u ityendezwa i nomsoor vea kwase ve er u v...  \n",
       "2  Ityendezwa i̱ nomsoor vea kwase ve er ér vea v...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)             \n",
    "with open(target_file) as f:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M_2ouEOH1_1q"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n"
     ]
    }
   ],
   "source": [
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z_1BwAApEtMk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fuzzywuzzy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (0.17.0)\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: python-Levenshtein in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (0.12.0)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from python-Levenshtein) (45.2.0.post20200210)\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "from os import cpu_count\n",
    "from functools import partial\n",
    "from multiprocessing import Pool\n",
    "\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_pp.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇩']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done in 0.0h:18.0min:34.256463050842285seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "### iterating over pandas dataframe rows is not recomended, let use multi processing to apply the function\n",
    "\n",
    "with Pool(cpu_count()-1) as pool:\n",
    "    scores = pool.map(partial(fuzzfilter, candidates=list(en_test_sents), pad=5), df_pp['source_sentence'])\n",
    "hours, rem = divmod(time.time() - start_time, 3600)\n",
    "minutes, seconds = divmod(rem, 60)\n",
    "print(\"done in {}h:{}min:{}seconds\".format(hours, minutes, seconds))\n",
    "\n",
    "# Filter out \"almost overlapping samples\"\n",
    "df_pp = df_pp.assign(scores=scores)\n",
    "df_pp = df_pp[df_pp['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hxxBOCA-xXhy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "How we say something can be as important as what we say .\n",
      "All of them in wisdom you have made . The earth is full of your productions . ”\n",
      "The surviving righteous ones will see that God’s prophetic word is true .\n",
      "Take a lesson from the lilies of the field , how they are growing ; they do not toil , nor do they spin ; but I say to you that not even Solomon in all his glory was arrayed as one of these . ” ​ — Matthew 6 : 28 , 29 .\n",
      "Now that some time has elapsed since that change was introduced , we can ask ourselves : ‘ Am I using the time made available to have a Family Worship evening or to engage in personal study ?\n",
      "What adjustments did one young couple make , and with what results ?\n",
      "Such promises are practical because they enable those who put faith in them to face the future with hope and confidence . ​ — Hebrews 11 : 6 .\n",
      "A step toward good study habits is regularly to set aside time for Bible study .\n",
      "For some weeks , the prophet Elijah has been a guest of the widow of Zarephath , living in a chamber on her roof .\n",
      "To these also Jehovah’s chosen Servant brings a righteous standing before Jehovah .\n",
      "\n",
      "==> train.tiv <==\n",
      "Gbenda u se er kwagh la ngu a inja vough er kwaghôron wase la nahan .\n",
      "TERE , ityom You ngi atô kposo kposo , U er i cii sha kwaghfan , tar iv a akaa a U gbe la . ”\n",
      "Mbaperapera mba vea war la vea nenge e̱r kwaghôron u Aôndo u profeti a lu u mimi yô .\n",
      "Nenge nen er ityombo i ken toho i vesen yô ; ngi eren tom ga , mou kpaa ngi te ga , kpa M kaan ne ne , Solomon a iengem na cii je kpaa , mayange haa akondo er mô ve môm nahan ga . ” — Mateu 6 : 28 , 29 .\n",
      "Er yange se gema kwagh ne , shighe a we karen hegen ne , se fatyô u pinen ayol a ase ser : ‘ M ngu eren Mcivir u hen Tsombor u aikighe la shin m ngu henen kwagh sha tseeneke wam a shighe ne kpa ?\n",
      "Nom man kwase ugen yange ve sôr akaa ken uma ve nena , man kwagh ne va a nyi ?\n",
      "( Pasalmi 72 : 12 , 13 ) Uityendezwa mban doo , sha ci u ka ve wase ior mba ve ne ve jighjigh la u lun a ishimaveren ér guda ngu van . — Mbaheberu 11 : 6 .\n",
      "U veren shighe u henen Bibilo hanma shighe la ka kwagh u vesen sha u lun a ieren i dedoo i henen kwagh .\n",
      "Profeti Eliya lu hen ya u kwasecôghol u ken Sarefati usati imôngo , lu tsan sha iyou i yaven i sha .\n",
      "Mban kpa , Wanakiriki u Yehova na , ve hingir mbaperapera sha ishigh ki Yehova .\n",
      "==> dev.en <==\n",
      "Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "It could be that an individual is acting in a way that is not appropriate .\n",
      "What wrong conclusion might some draw when thinking about showing kindness to others ?\n",
      "Death is like sleep in that the dead are unconscious and cannot do anything .\n",
      "Who can say that he has totally freed himself from walking in accord with the flesh ?\n",
      "We read : “ Abraham got up early in the morning and saddled his ass and took two of his attendants with him and Isaac his son ; and he split the wood for the burnt offering .\n",
      "Grief can affect us in various ways , but for many the overriding feeling is one of intense emotional pain .\n",
      "We will look for them as soon as we get there . ”\n",
      "\n",
      "==> dev.tiv <==\n",
      "10 : ​ 6 - 9 .\n",
      "Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "Ka mnenge u kpeegh u nyi nahan mbagenev ve lu a mi sha kwagh u erdoo ?\n",
      "Ku ngu er mnyam nahan , sha ci u mbakpenev kera fa ma kwagh ga , shi vea fatyô u eren kwagh môm ga .\n",
      "Ka an nana fatyô u kaan ér nan ngu iyol ishô elegh elegh sha kwagh u zenden sha asaren a iyolougho ?\n",
      "Bibilo kaa ér : “ Tsô Aberaham mough pepe pepe kange jaki na shidi ; a mough a mbayev nav uhar kua Isaka wan na ; a ar ikyon sha ci u nagh ku nanden ; a gba zan hen ijiir i Aôndo tese un la . ”\n",
      "Ijungwen ngi benden a vese sha igbenda kposo kposo , kpa mbagenev yô , ka i na ve mnyoon ken ishima tsung .\n",
      "Se za sôôn a sôôn ken gar u Mexico City nahan maa se tôv ve ityô je . ”\n"
     ]
    }
   ],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iBRMm4kMxZ8L"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'joeynmt' already exists and is not an empty directory.\n",
      "Processing /home/ec2-user/SageMaker/masakhane/joeynmt\n",
      "Requirement already satisfied: future in /usr/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.17.1)\n",
      "Requirement already satisfied: pillow in /usr/lib64/python3.6/dist-packages (from joeynmt==0.0.1) (6.2.1)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/lib64/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.4)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/lib/python3.6/dist-packages (from joeynmt==0.0.1) (41.6.0)\n",
      "Collecting torch>=1.1 (from joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/24/19/4804aea17cd136f1705a5e98a00618cb8f6ccc375ad8bfa437408e09d058/torch-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (753.4MB)\n",
      "\u001b[K    100% |████████████████████████████████| 753.4MB 64kB/s  eta 0:00:01    21% |██████▊                         | 158.7MB 57.5MB/s eta 0:00:11    33% |██████████▊                     | 253.7MB 46.6MB/s eta 0:00:11    99% |███████████████████████████████▊| 745.9MB 56.9MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting tensorflow>=1.14 (from joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/85/d4/c0cd1057b331bc38b65478302114194bd8e1b9c2bbc06e300935c0e93d90/tensorflow-2.1.0-cp36-cp36m-manylinux2010_x86_64.whl (421.8MB)\n",
      "\u001b[K    100% |████████████████████████████████| 421.8MB 116kB/s eta 0:00:01    47% |███████████████▏                | 200.2MB 60.7MB/s eta 0:00:04\n",
      "\u001b[?25hCollecting torchtext (from joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/79/ef/54b8da26f37787f5c670ae2199329e7dccf195c060b25628d99e587dac51/torchtext-0.5.0-py3-none-any.whl (73kB)\n",
      "\u001b[K    100% |████████████████████████████████| 81kB 34.4MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting sacrebleu>=1.3.6 (from joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/45/31/1a135b964c169984b27fb2f7a50280fa7f8e6d9d404d8a9e596180487fd1/sacrebleu-1.4.3-py3-none-any.whl\n",
      "Requirement already satisfied: subword-nmt in /usr/local/lib/python3.6/site-packages (from joeynmt==0.0.1) (0.3.7)\n",
      "Requirement already satisfied: matplotlib in /usr/lib64/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.1)\n",
      "Collecting seaborn (from joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/70/bd/5e6bf595fe6ee0f257ae49336dd180768c1ed3d7c7155b2fdf894c1c808a/seaborn-0.10.0-py3-none-any.whl (215kB)\n",
      "\u001b[K    100% |████████████████████████████████| 225kB 38.4MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting pyyaml>=5.1 (from joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)\n",
      "\u001b[K    100% |████████████████████████████████| 276kB 36.0MB/s ta 0:00:01��█████████████████████▋      | 215kB 63.2MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pylint in /usr/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.9.4)\n",
      "Collecting six==1.12 (from joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Collecting google-pasta>=0.1.6 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c3/fd/1e86bc4837cc9a3a5faf3db9b1854aa04ad35b5f381f9648fbe81a6f94e4/google_pasta-0.1.8-py3-none-any.whl (57kB)\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 18.0MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting opt-einsum>=2.3.2 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b8/83/755bd5324777875e9dff19c2e59daec837d0378c09196634524a3d7269ac/opt_einsum-3.1.0.tar.gz (69kB)\n",
      "\u001b[K    100% |████████████████████████████████| 71kB 32.0MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/lib64/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
      "Collecting grpcio>=1.8.6 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/45/6d/ee60b05e5e6df8d27561bcb2579b4b2e4004561be7c843768359e005093b/grpcio-1.27.1-cp36-cp36m-manylinux2010_x86_64.whl (2.7MB)\n",
      "\u001b[K    100% |████████████████████████████████| 2.7MB 14.2MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.32.3)\n",
      "Collecting scipy==1.4.1; python_version >= \"3\" (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Using cached https://files.pythonhosted.org/packages/dc/29/162476fd44203116e7980cfbd9352eef9db37c49445d1fec35509022f6aa/scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl\n",
      "Collecting astor>=0.6.0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/c3/88/97eef84f48fa04fbd6750e62dcceafba6c63c81b7ac1420856c8dcc0a3f9/astor-0.8.1-py2.py3-none-any.whl\n",
      "Collecting tensorboard<2.2.0,>=2.1.0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/40/23/53ffe290341cd0855d595b0a2e7485932f473798af173bbe3a584b99bb06/tensorboard-2.1.0-py3-none-any.whl (3.8MB)\n",
      "\u001b[K    100% |████████████████████████████████| 3.8MB 10.6MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting termcolor>=1.1.0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz\n",
      "Collecting tensorflow-estimator<2.2.0,>=2.1.0rc0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/18/90/b77c328a1304437ab1310b463e533fa7689f4bfc41549593056d812fab8e/tensorflow_estimator-2.1.0-py2.py3-none-any.whl (448kB)\n",
      "\u001b[K    100% |████████████████████████████████| 450kB 33.1MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting gast==0.2.2 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n",
      "Collecting keras-preprocessing>=1.1.0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41kB)\n",
      "\u001b[K    100% |████████████████████████████████| 51kB 17.2MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting keras-applications>=1.0.8 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)\n",
      "\u001b[K    100% |████████████████████████████████| 51kB 24.8MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: protobuf>=3.8.0 in /usr/lib64/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
      "Collecting absl-py>=0.7.0 (from tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1a/53/9243c600e047bd4c3df9e69cfabc1e8004a82cac2e0c484580a78a94ba2a/absl-py-0.9.0.tar.gz (104kB)\n",
      "\u001b[K    100% |████████████████████████████████| 112kB 41.5MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting tqdm (from torchtext->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/cd/80/5bb262050dd2f30f8819626b7c92339708fe2ed7bd5554c8193b4487b367/tqdm-4.42.1-py2.py3-none-any.whl (59kB)\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 29.0MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: requests in /usr/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.20.0)\n",
      "Collecting sentencepiece (from torchtext->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/74/f4/2d5214cbf13d06e7cb2c20d84115ca25b53ea76fa1f0ade0e3c9749de214/sentencepiece-0.1.85-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)\n",
      "\u001b[K    100% |████████████████████████████████| 1.0MB 25.8MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting typing (from sacrebleu>=1.3.6->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/fe/2e/b480ee1b75e6d17d2993738670e75c1feeb9ff7f64452153cf018051cc92/typing-3.7.4.1-py3-none-any.whl\n",
      "Collecting portalocker (from sacrebleu>=1.3.6->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.5)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.8.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/lib64/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
      "Requirement already satisfied: pandas>=0.22.0 in /usr/lib64/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.24.2)\n",
      "Requirement already satisfied: isort>=4.2.5 in /usr/lib/python3.6/dist-packages (from pylint->joeynmt==0.0.1) (4.3.21)\n",
      "Requirement already satisfied: mccabe in /usr/lib/python3.6/dist-packages (from pylint->joeynmt==0.0.1) (0.6.1)\n",
      "Requirement already satisfied: astroid<2.0,>=1.6 in /usr/lib/python3.6/dist-packages (from pylint->joeynmt==0.0.1) (1.6.6)\n",
      "Collecting werkzeug>=0.11.15 (from tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ba/a5/d6f8a6e71f15364d35678a4ec8a0186f980b3bd2545f40ad51dd26a87fb1/Werkzeug-1.0.0-py2.py3-none-any.whl (298kB)\n",
      "\u001b[K    100% |████████████████████████████████| 307kB 40.9MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting markdown>=2.6.8 (from tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d5/16/c5a68ef8c62406b3bbd8f49199bbae56feb390746a284c4cf036c687465f/Markdown-3.2-py2.py3-none-any.whl (88kB)\n",
      "\u001b[K    100% |████████████████████████████████| 92kB 37.9MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting google-auth-oauthlib<0.5,>=0.4.1 (from tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/7b/b8/88def36e74bee9fce511c9519571f4e485e890093ab7442284f4ffaef60b/google_auth_oauthlib-0.4.1-py2.py3-none-any.whl\n",
      "Collecting google-auth<2,>=1.6.3 (from tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1c/6d/7aae38a9022f982cf8167775c7fc299f203417b698c27080ce09060bba07/google_auth-1.11.0-py2.py3-none-any.whl (76kB)\n",
      "\u001b[K    100% |████████████████████████████████| 81kB 35.4MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting h5py (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/60/06/cafdd44889200e5438b897388f3075b52a8ef01f28a17366d91de0fa2d05/h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9MB)\n",
      "\u001b[K    100% |████████████████████████████████| 2.9MB 15.6MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: idna<2.8,>=2.5 in /usr/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.7)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.9.11)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
      "Requirement already satisfied: pytz>=2011k in /usr/lib/python3.6/dist-packages (from pandas>=0.22.0->seaborn->joeynmt==0.0.1) (2019.3)\n",
      "Requirement already satisfied: lazy-object-proxy in /usr/lib64/python3.6/dist-packages (from astroid<2.0,>=1.6->pylint->joeynmt==0.0.1) (1.4.3)\n",
      "Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl\n",
      "Collecting cachetools<5.0,>=2.0.0 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "  Downloading https://files.pythonhosted.org/packages/08/6a/abf83cb951617793fd49c98cb9456860f5df66ff89883c8660aa0672d425/cachetools-4.0.0-py3-none-any.whl\n",
      "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1) (3.4.2)\n",
      "Collecting pyasn1-modules>=0.2.1 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155kB)\n",
      "\u001b[K    100% |████████████████████████████████| 163kB 42.0MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)\n",
      "\u001b[K    100% |████████████████████████████████| 153kB 41.2MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3.6/dist-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.8)\n",
      "Building wheels for collected packages: joeynmt, pyyaml, opt-einsum, termcolor, gast, absl-py\n",
      "  Building wheel for joeynmt (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /tmp/pip-ephem-wheel-cache-g8wwbcp9/wheels/3a/30/44/d8a7e4bf0ed7c54548bf491738b185ef57222ad3bcdfb1afc4\n",
      "  Building wheel for pyyaml (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/ec2-user/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191\n",
      "  Building wheel for opt-einsum (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/ec2-user/.cache/pip/wheels/2c/b1/94/43d03e130b929aae7ba3f8d15cbd7bc0d1cb5bb38a5c721833\n",
      "  Building wheel for termcolor (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/ec2-user/.cache/pip/wheels/7c/06/54/bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6\n",
      "  Building wheel for gast (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/ec2-user/.cache/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd\n",
      "  Building wheel for absl-py (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/ec2-user/.cache/pip/wheels/8e/28/49/fad4e7f0b9a1227708cbbee4487ac8558a7334849cb81c813d\n",
      "Successfully built joeynmt pyyaml opt-einsum termcolor gast absl-py\n",
      "\u001b[31mcoremltools 2.0 has requirement six==1.10.0, but you'll have six 1.12.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mboto3 1.9.72 has requirement botocore<1.13.0,>=1.12.72, but you'll have botocore 1.12.66 which is incompatible.\u001b[0m\n",
      "\u001b[31mawscli 1.16.76 has requirement PyYAML<=3.13,>=3.10, but you'll have pyyaml 5.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mtensorboard 2.1.0 has requirement requests<3,>=2.21.0, but you'll have requests 2.20.0 which is incompatible.\u001b[0m\n",
      "Installing collected packages: torch, six, google-pasta, opt-einsum, grpcio, scipy, astor, werkzeug, markdown, cachetools, pyasn1-modules, google-auth, oauthlib, requests-oauthlib, google-auth-oauthlib, absl-py, tensorboard, termcolor, tensorflow-estimator, gast, keras-preprocessing, h5py, keras-applications, tensorflow, tqdm, sentencepiece, torchtext, typing, portalocker, sacrebleu, seaborn, pyyaml, joeynmt\n",
      "\u001b[31mCould not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/lib64/python3.6/dist-packages/torch-1.4.0.dist-info'\n",
      "Consider using the `--user` option or check the permissions.\n",
      "\u001b[0m\n",
      "\u001b[33mYou are using pip version 19.0.2, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip3 install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: subword-nmt in /usr/local/lib/python3.6/site-packages (0.3.7)\n",
      "\u001b[33mYou are using pip version 19.0.2, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!sudo pip3 install subword-nmt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H-TyjtmXB1mL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.tiv    test.tiv\t    train.en\n",
      "dev.bpe.en\tdev.tiv      test.en\t     train.bpe.en   train.tiv\n",
      "dev.bpe.tiv\ttest.bpe.en  test.en-any.en  train.bpe.tiv\n",
      "cp: cannot create regular file ‘’: No such file or directory\n",
      "dev.bpe.en   dev.tiv\t   test.en\t   train.bpe.en   train.tiv\n",
      "dev.bpe.tiv  test.bpe.en   test.en-any.en  train.bpe.tiv\n",
      "dev.en\t     test.bpe.tiv  test.tiv\t   train.en\n",
      "BPE Tiv Sentences\n",
      "Er nan ve yange gba u H@@ u@@ sh@@ ai una lu a ishimataver keng ve una fatyô u civir Aôndo sha mimi ?\n",
      "Er nan ve i gbe u se lu a ishimataver ve se fatyô u civir Aôndo sha mimi ?\n",
      "M sôn Yehova mer a wasem me taver ishima me er kwagh u m tsough u eren la .\n",
      "Hegen ve gema inja , nahan ka m z@@ aan ve inya hanma shighe . ” — Ôr Anzaakaa 29 : 25 .\n",
      "[ 1 ] ( ikyum@@ hi@@ ange i sha 7 ) I gema ati agen .\n",
      "Combined BPE Vocab\n",
      "erus@@\n",
      "isholi@@\n",
      "š@@\n",
      "epher@@\n",
      "ị\n",
      "î@@\n",
      "Ó@@\n",
      "ö\n",
      "_\n",
      "ateu\n"
     ]
    }
   ],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$experiment_path\"\n",
    "! cp test.* \"$experiment_path\"\n",
    "! cp dev.* \"$experiment_path\"\n",
    "! cp bpe.codes.4000 \"$gexperiment_path\"\n",
    "! ls \"$experiment_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Tiv Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIs1lY2hxMsl"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (source_language, target_language)\n",
    "gdrive_path = os.environ[\"experiment_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{experiment_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 150                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_transformer\"\n",
    "    overwrite: True               # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 5\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, experiment_path=os.environ[\"experiment_path\"], source_language=source_language, target_language=target_language)\n",
    "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.5.12\n",
      "  latest version: 4.8.2\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ec2-user/anaconda3/envs/python3\n",
      "\n",
      "  added / updated specs: \n",
      "    - cudatoolkit=10.1\n",
      "    - pytorch\n",
      "    - torchvision\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    certifi-2019.11.28         |           py36_0         156 KB\n",
      "    ca-certificates-2020.1.1   |                0         132 KB\n",
      "    openssl-1.0.2u             |       h7b6447c_0         3.1 MB\n",
      "    pytorch-1.4.0              |py3.6_cuda10.1.243_cudnn7.6.3_0       432.9 MB  pytorch\n",
      "    torchvision-0.5.0          |       py36_cu101         9.1 MB  pytorch\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:       445.4 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    cudatoolkit:     10.1.243-h6bb024c_0                          \n",
      "    ninja:           1.9.0-py36hfd86e86_0                         \n",
      "    pytorch:         1.4.0-py3.6_cuda10.1.243_cudnn7.6.3_0 pytorch\n",
      "    torchvision:     0.5.0-py36_cu101                      pytorch\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    ca-certificates: 2019.10.16-0                                  --> 2020.1.1-0       \n",
      "    certifi:         2019.9.11-py36_0                              --> 2019.11.28-py36_0\n",
      "    openssl:         1.0.2t-h7b6447c_1                             --> 1.0.2u-h7b6447c_0\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "certifi-2019.11.28   | 156 KB    | ##################################### | 100% \n",
      "ca-certificates-2020 | 132 KB    | ##################################### | 100% \n",
      "openssl-1.0.2u       | 3.1 MB    | ##################################### | 100% \n",
      "pytorch-1.4.0        | 432.9 MB  | ##################################### | 100% \n",
      "torchvision-0.5.0    | 9.1 MB    | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.5.12\n",
      "  latest version: 4.8.2\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ec2-user/anaconda3/envs/python3\n",
      "\n",
      "  added / updated specs: \n",
      "    - tensorboard\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    tensorboard-2.0.0          |     pyhb38c66f_1         3.3 MB\n",
      "    joblib-0.14.1              |             py_0         202 KB\n",
      "    numexpr-2.7.1              |   py36h423224d_0         197 KB\n",
      "    six-1.14.0                 |           py36_0          27 KB\n",
      "    mkl-2020.0                 |              166       202.1 MB\n",
      "    markdown-3.1.1             |           py36_0         113 KB\n",
      "    c-ares-1.15.0              |    h7b6447c_1001         102 KB\n",
      "    numpy-1.18.1               |   py36h4f9e942_0           5 KB\n",
      "    setuptools-45.2.0          |           py36_0         659 KB\n",
      "    absl-py-0.8.1              |           py36_0         161 KB\n",
      "    numpy-base-1.18.1          |   py36hde5b4d6_1         5.2 MB\n",
      "    scikit-learn-0.22.1        |   py36hd81dba3_0         7.1 MB\n",
      "    scipy-1.4.1                |   py36h0b6359f_0        18.9 MB\n",
      "    grpcio-1.14.1              |   py36h9ba97e2_0         1.0 MB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:       239.1 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    absl-py:      0.8.1-py36_0         \n",
      "    c-ares:       1.15.0-h7b6447c_1001 \n",
      "    grpcio:       1.14.1-py36h9ba97e2_0\n",
      "    joblib:       0.14.1-py_0          \n",
      "    markdown:     3.1.1-py36_0         \n",
      "    tensorboard:  2.0.0-pyhb38c66f_1   \n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    mkl:          2018.0.2-1            --> 2020.0-166           \n",
      "    mkl-service:  1.1.2-py36h17a0993_4  --> 2.3.0-py36he904b0f_0 \n",
      "    numexpr:      2.6.5-py36h7bf3b9c_0  --> 2.7.1-py36h423224d_0 \n",
      "    numpy:        1.14.3-py36hcd700cb_1 --> 1.18.1-py36h4f9e942_0\n",
      "    numpy-base:   1.14.3-py36h9be14a7_1 --> 1.18.1-py36hde5b4d6_1\n",
      "    scikit-learn: 0.19.1-py36h7aa7ec6_0 --> 0.22.1-py36hd81dba3_0\n",
      "    scipy:        1.1.0-py36hfc37229_0  --> 1.4.1-py36h0b6359f_0 \n",
      "    setuptools:   39.1.0-py36_0         --> 45.2.0-py36_0        \n",
      "    six:          1.11.0-py36h372c433_1 --> 1.14.0-py36_0        \n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "tensorboard-2.0.0    | 3.3 MB    | ##################################### | 100% \n",
      "joblib-0.14.1        | 202 KB    | ##################################### | 100% \n",
      "numexpr-2.7.1        | 197 KB    | ##################################### | 100% \n",
      "six-1.14.0           | 27 KB     | ##################################### | 100% \n",
      "mkl-2020.0           | 202.1 MB  | ##################################### | 100% \n",
      "markdown-3.1.1       | 113 KB    | ##################################### | 100% \n",
      "c-ares-1.15.0        | 102 KB    | ##################################### | 100% \n",
      "numpy-1.18.1         | 5 KB      | ##################################### | 100% \n",
      "setuptools-45.2.0    | 659 KB    | ##################################### | 100% \n",
      "absl-py-0.8.1        | 161 KB    | ##################################### | 100% \n",
      "numpy-base-1.18.1    | 5.2 MB    | ##################################### | 100% \n",
      "scikit-learn-0.22.1  | 7.1 MB    | ##################################### | 100% \n",
      "scipy-1.4.1          | 18.9 MB   | ##################################### | 100% \n",
      "grpcio-1.14.1        | 1.0 MB    | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.5.12\n",
      "  latest version: 4.8.2\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ec2-user/anaconda3/envs/python3\n",
      "\n",
      "  added / updated specs: \n",
      "    - torchtext\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    torchtext-0.5.0            |             py_1         1.4 MB  pytorch\n",
      "    tqdm-4.42.1                |             py_0          56 KB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:         1.5 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    torchtext: 0.5.0-py_1  pytorch\n",
      "    tqdm:      4.42.1-py_0        \n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "torchtext-0.5.0      | 1.4 MB    | ##################################### | 100% \n",
      "tqdm-4.42.1          | 56 KB     | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.5.12\n",
      "  latest version: 4.8.2\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ec2-user/anaconda3/envs/python3\n",
      "\n",
      "  added / updated specs: \n",
      "    - sentencepiece\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    sentencepiece-0.1.84       |   py36h6bb024c_0         3.1 MB  powerai\n",
      "    tensorflow-base-2.0.0      |mkl_py36h9204916_0       100.9 MB\n",
      "    termcolor-1.1.0            |           py36_1           7 KB\n",
      "    _tflow_select-2.3.0        |              mkl           2 KB\n",
      "    google-pasta-0.1.8         |             py_0          43 KB\n",
      "    keras-preprocessing-1.1.0  |             py_1          36 KB\n",
      "    tensorflow-estimator-2.0.0 |     pyh2649769_0         272 KB\n",
      "    tensorflow-2.0.0           |mkl_py36hef7ec59_0           3 KB\n",
      "    gast-0.2.2                 |           py36_0         138 KB\n",
      "    keras-applications-1.0.8   |             py_0          33 KB\n",
      "    astor-0.8.0                |           py36_0          45 KB\n",
      "    opt_einsum-3.1.0           |             py_0          54 KB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:       104.6 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    _tflow_select:        2.3.0-mkl                       \n",
      "    astor:                0.8.0-py36_0                    \n",
      "    gast:                 0.2.2-py36_0                    \n",
      "    google-pasta:         0.1.8-py_0                      \n",
      "    keras-applications:   1.0.8-py_0                      \n",
      "    keras-preprocessing:  1.1.0-py_1                      \n",
      "    opt_einsum:           3.1.0-py_0                      \n",
      "    sentencepiece:        0.1.84-py36h6bb024c_0    powerai\n",
      "    tensorflow:           2.0.0-mkl_py36hef7ec59_0        \n",
      "    tensorflow-base:      2.0.0-mkl_py36h9204916_0        \n",
      "    tensorflow-estimator: 2.0.0-pyh2649769_0              \n",
      "    termcolor:            1.1.0-py36_1                    \n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    wrapt:                1.10.11-py36h28b7045_0           --> 1.11.2-py36h7b6447c_0\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentencepiece-0.1.84 | 3.1 MB    | ##################################### | 100% \n",
      "tensorflow-base-2.0. | 100.9 MB  | ##################################### | 100% \n",
      "termcolor-1.1.0      | 7 KB      | ##################################### | 100% \n",
      "_tflow_select-2.3.0  | 2 KB      | ##################################### | 100% \n",
      "google-pasta-0.1.8   | 43 KB     | ##################################### | 100% \n",
      "keras-preprocessing- | 36 KB     | ##################################### | 100% \n",
      "tensorflow-estimator | 272 KB    | ##################################### | 100% \n",
      "tensorflow-2.0.0     | 3 KB      | ##################################### | 100% \n",
      "gast-0.2.2           | 138 KB    | ##################################### | 100% \n",
      "keras-applications-1 | 33 KB     | ##################################### | 100% \n",
      "astor-0.8.0          | 45 KB     | ##################################### | 100% \n",
      "opt_einsum-3.1.0     | 54 KB     | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.5.12\n",
      "  latest version: 4.8.2\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ec2-user/anaconda3/envs/python3\n",
      "\n",
      "  added / updated specs: \n",
      "    - sacrebleu\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    portalocker-1.5.2          |           py36_0          20 KB\n",
      "    sacrebleu-1.4.3            |             py_0          34 KB  powerai\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:          54 KB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    portalocker: 1.5.2-py36_0        \n",
      "    sacrebleu:   1.4.3-py_0   powerai\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "portalocker-1.5.2    | 20 KB     | ##################################### | 100% \n",
      "sacrebleu-1.4.3      | 34 KB     | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n"
     ]
    }
   ],
   "source": [
    "!conda install pytorch torchvision cudatoolkit=10.1 -c pytorch --yes\n",
    "!conda install tensorboard --yes\n",
    "!conda install -c pytorch torchtext --yes\n",
    "!conda install -c powerai sentencepiece --yes\n",
    "!conda install -c powerai sacrebleu --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pIifxE3Qzuvs"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6ZBPFwT94WpI"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 15:55:50,862 Hello! This is Joey-NMT.\n",
      "2020-02-12 15:55:52,201 Total params: 12152320\n",
      "2020-02-12 15:55:52,202 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 15:55:55,584 cfg.name                           : entiv_transformer\n",
      "2020-02-12 15:55:55,584 cfg.data.src                       : en\n",
      "2020-02-12 15:55:55,584 cfg.data.trg                       : tiv\n",
      "2020-02-12 15:55:55,584 cfg.data.train                     : data/entiv/train.bpe\n",
      "2020-02-12 15:55:55,585 cfg.data.dev                       : data/entiv/dev.bpe\n",
      "2020-02-12 15:55:55,585 cfg.data.test                      : data/entiv/test.bpe\n",
      "2020-02-12 15:55:55,585 cfg.data.level                     : bpe\n",
      "2020-02-12 15:55:55,585 cfg.data.lowercase                 : False\n",
      "2020-02-12 15:55:55,585 cfg.data.max_sent_length           : 100\n",
      "2020-02-12 15:55:55,585 cfg.data.src_vocab                 : data/entiv/vocab.txt\n",
      "2020-02-12 15:55:55,585 cfg.data.trg_vocab                 : data/entiv/vocab.txt\n",
      "2020-02-12 15:55:55,585 cfg.testing.beam_size              : 5\n",
      "2020-02-12 15:55:55,585 cfg.testing.alpha                  : 1.0\n",
      "2020-02-12 15:55:55,585 cfg.training.random_seed           : 42\n",
      "2020-02-12 15:55:55,585 cfg.training.optimizer             : adam\n",
      "2020-02-12 15:55:55,585 cfg.training.normalization         : tokens\n",
      "2020-02-12 15:55:55,585 cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2020-02-12 15:55:55,585 cfg.training.scheduling            : plateau\n",
      "2020-02-12 15:55:55,585 cfg.training.patience              : 5\n",
      "2020-02-12 15:55:55,586 cfg.training.learning_rate_factor  : 0.5\n",
      "2020-02-12 15:55:55,586 cfg.training.learning_rate_warmup  : 1000\n",
      "2020-02-12 15:55:55,586 cfg.training.decrease_factor       : 0.7\n",
      "2020-02-12 15:55:55,586 cfg.training.loss                  : crossentropy\n",
      "2020-02-12 15:55:55,586 cfg.training.learning_rate         : 0.0003\n",
      "2020-02-12 15:55:55,586 cfg.training.learning_rate_min     : 1e-08\n",
      "2020-02-12 15:55:55,586 cfg.training.weight_decay          : 0.0\n",
      "2020-02-12 15:55:55,586 cfg.training.label_smoothing       : 0.1\n",
      "2020-02-12 15:55:55,586 cfg.training.batch_size            : 4096\n",
      "2020-02-12 15:55:55,586 cfg.training.batch_type            : token\n",
      "2020-02-12 15:55:55,586 cfg.training.eval_batch_size       : 3600\n",
      "2020-02-12 15:55:55,586 cfg.training.eval_batch_type       : token\n",
      "2020-02-12 15:55:55,586 cfg.training.batch_multiplier      : 1\n",
      "2020-02-12 15:55:55,586 cfg.training.early_stopping_metric : ppl\n",
      "2020-02-12 15:55:55,586 cfg.training.epochs                : 150\n",
      "2020-02-12 15:55:55,586 cfg.training.validation_freq       : 1000\n",
      "2020-02-12 15:55:55,587 cfg.training.logging_freq          : 100\n",
      "2020-02-12 15:55:55,587 cfg.training.eval_metric           : bleu\n",
      "2020-02-12 15:55:55,587 cfg.training.model_dir             : models/entiv_transformer\n",
      "2020-02-12 15:55:55,587 cfg.training.overwrite             : True\n",
      "2020-02-12 15:55:55,587 cfg.training.shuffle               : True\n",
      "2020-02-12 15:55:55,587 cfg.training.use_cuda              : True\n",
      "2020-02-12 15:55:55,587 cfg.training.max_output_length     : 100\n",
      "2020-02-12 15:55:55,587 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2020-02-12 15:55:55,587 cfg.training.keep_last_ckpts       : 5\n",
      "2020-02-12 15:55:55,587 cfg.model.initializer              : xavier\n",
      "2020-02-12 15:55:55,587 cfg.model.bias_initializer         : zeros\n",
      "2020-02-12 15:55:55,587 cfg.model.init_gain                : 1.0\n",
      "2020-02-12 15:55:55,587 cfg.model.embed_initializer        : xavier\n",
      "2020-02-12 15:55:55,587 cfg.model.embed_init_gain          : 1.0\n",
      "2020-02-12 15:55:55,587 cfg.model.tied_embeddings          : True\n",
      "2020-02-12 15:55:55,588 cfg.model.tied_softmax             : True\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.type             : transformer\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.num_layers       : 6\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.num_heads        : 4\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.embeddings.scale : True\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.hidden_size      : 256\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.ff_size          : 1024\n",
      "2020-02-12 15:55:55,588 cfg.model.encoder.dropout          : 0.3\n",
      "2020-02-12 15:55:55,588 cfg.model.decoder.type             : transformer\n",
      "2020-02-12 15:55:55,588 cfg.model.decoder.num_layers       : 6\n",
      "2020-02-12 15:55:55,588 cfg.model.decoder.num_heads        : 4\n",
      "2020-02-12 15:55:55,588 cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2020-02-12 15:55:55,588 cfg.model.decoder.embeddings.scale : True\n",
      "2020-02-12 15:55:55,589 cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2020-02-12 15:55:55,589 cfg.model.decoder.hidden_size      : 256\n",
      "2020-02-12 15:55:55,589 cfg.model.decoder.ff_size          : 1024\n",
      "2020-02-12 15:55:55,589 cfg.model.decoder.dropout          : 0.3\n",
      "2020-02-12 15:55:55,589 Data set sizes: \n",
      "\ttrain 185707,\n",
      "\tvalid 1000,\n",
      "\ttest 375\n",
      "2020-02-12 15:55:55,589 First training example:\n",
      "\t[SRC] How we say something can be as important as what we say .\n",
      "\t[TRG] Gbenda u se er kwagh la ngu a inja vough er kwaghôron wase la nahan .\n",
      "2020-02-12 15:55:55,589 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) u (7) a (8) the (9) i\n",
      "2020-02-12 15:55:55,590 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) u (7) a (8) the (9) i\n",
      "2020-02-12 15:55:55,590 Number of Src words (types): 4266\n",
      "2020-02-12 15:55:55,590 Number of Trg words (types): 4266\n",
      "2020-02-12 15:55:55,590 Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4266),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4266))\n",
      "2020-02-12 15:55:55,595 EPOCH 1\n",
      "2020-02-12 15:56:06,328 Epoch   1 Step:      100 Batch Loss:     5.058401 Tokens per Sec:    20734, Lr: 0.000300\n",
      "2020-02-12 15:56:16,422 Epoch   1 Step:      200 Batch Loss:     4.870168 Tokens per Sec:    21861, Lr: 0.000300\n",
      "2020-02-12 15:56:26,533 Epoch   1 Step:      300 Batch Loss:     4.652421 Tokens per Sec:    22052, Lr: 0.000300\n",
      "2020-02-12 15:56:36,611 Epoch   1 Step:      400 Batch Loss:     4.356110 Tokens per Sec:    21940, Lr: 0.000300\n",
      "2020-02-12 15:56:46,722 Epoch   1 Step:      500 Batch Loss:     4.122426 Tokens per Sec:    21766, Lr: 0.000300\n",
      "2020-02-12 15:56:56,734 Epoch   1 Step:      600 Batch Loss:     4.076796 Tokens per Sec:    21913, Lr: 0.000300\n",
      "2020-02-12 15:57:06,588 Epoch   1 Step:      700 Batch Loss:     4.210296 Tokens per Sec:    22534, Lr: 0.000300\n",
      "2020-02-12 15:57:16,444 Epoch   1 Step:      800 Batch Loss:     3.956947 Tokens per Sec:    22428, Lr: 0.000300\n",
      "2020-02-12 15:57:26,470 Epoch   1 Step:      900 Batch Loss:     3.769588 Tokens per Sec:    22100, Lr: 0.000300\n",
      "2020-02-12 15:57:36,389 Epoch   1 Step:     1000 Batch Loss:     3.811076 Tokens per Sec:    22413, Lr: 0.000300\n",
      "2020-02-12 15:57:58,754 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 15:57:58,754 Saving new checkpoint.\n",
      "2020-02-12 15:57:58,924 Example #0\n",
      "2020-02-12 15:57:58,925 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 15:57:58,925 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 15:57:58,925 \tHypothesis: ( Mateu 4 : 1 ) Nahan kpa , se fatyô u se lu a mi sha ci u se lu a mi la , se lu a mi yô , se lu a mi sha ci u se lu a mi la .\n",
      "2020-02-12 15:57:58,925 Example #1\n",
      "2020-02-12 15:57:58,925 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 15:57:58,925 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 15:57:58,925 \tHypothesis: ( 1 Mbakorinte 1 : 1 ) Nahan kpa , se lu “ ishima i nan ve ishima i nan ve ve ishima i nan ve la .\n",
      "2020-02-12 15:57:58,925 Example #2\n",
      "2020-02-12 15:57:58,926 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 15:57:58,926 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 15:57:58,926 \tHypothesis: ( Mateu 3 : 1 ) Nahan kpa , m lu a mi yô , m lu a mi yô , m lu a mi .\n",
      "2020-02-12 15:57:58,926 Example #3\n",
      "2020-02-12 15:57:58,926 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 15:57:58,926 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 15:57:58,926 \tHypothesis: ( 1 Mbakorinte 4 : 1 ) Nahan kpa , se fatyô u se lu a mi yô , se fatyô u se lu a mi la .\n",
      "2020-02-12 15:57:58,926 Validation result (greedy) at epoch   1, step     1000: bleu:   2.24, loss: 105431.4141, ppl:  35.2121, duration: 22.5367s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 15:58:09,004 Epoch   1 Step:     1100 Batch Loss:     3.681341 Tokens per Sec:    21252, Lr: 0.000300\n",
      "2020-02-12 15:58:19,066 Epoch   1 Step:     1200 Batch Loss:     3.286341 Tokens per Sec:    22182, Lr: 0.000300\n",
      "2020-02-12 15:58:29,003 Epoch   1 Step:     1300 Batch Loss:     3.167710 Tokens per Sec:    22510, Lr: 0.000300\n",
      "2020-02-12 15:58:38,931 Epoch   1 Step:     1400 Batch Loss:     3.563505 Tokens per Sec:    22049, Lr: 0.000300\n",
      "2020-02-12 15:58:48,935 Epoch   1 Step:     1500 Batch Loss:     3.639344 Tokens per Sec:    22684, Lr: 0.000300\n",
      "2020-02-12 15:58:58,889 Epoch   1 Step:     1600 Batch Loss:     3.228676 Tokens per Sec:    22208, Lr: 0.000300\n",
      "2020-02-12 15:59:08,764 Epoch   1 Step:     1700 Batch Loss:     3.194575 Tokens per Sec:    21941, Lr: 0.000300\n",
      "2020-02-12 15:59:18,664 Epoch   1 Step:     1800 Batch Loss:     3.024248 Tokens per Sec:    22794, Lr: 0.000300\n",
      "2020-02-12 15:59:28,757 Epoch   1 Step:     1900 Batch Loss:     3.275215 Tokens per Sec:    21176, Lr: 0.000300\n",
      "2020-02-12 15:59:38,827 Epoch   1 Step:     2000 Batch Loss:     3.323348 Tokens per Sec:    22144, Lr: 0.000300\n",
      "2020-02-12 16:00:02,229 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:00:02,229 Saving new checkpoint.\n",
      "2020-02-12 16:00:02,400 Example #0\n",
      "2020-02-12 16:00:02,400 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:00:02,400 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:00:02,400 \tHypothesis: Er nan ve i gbe u se lu a mi sha u eren tom u Yehova a lu a mi la yô , ka u eren kwagh u a lu a mi la , shi a lu a mi sha ci u ve lu a mi la . — 1 : 1 , 4 .\n",
      "2020-02-12 16:00:02,400 Example #1\n",
      "2020-02-12 16:00:02,400 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:00:02,400 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:00:02,401 \tHypothesis: Se fatyô u ‘ eren msen ’ sha ci u se lu a mi la . — Ps .\n",
      "2020-02-12 16:00:02,401 Example #2\n",
      "2020-02-12 16:00:02,401 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:00:02,401 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:00:02,401 \tHypothesis: ( Mateu 9 : 9 ) Ken inyom i 199 la , i nger ken inyom i 1959 la , i lu ken inyom i 199 la .\n",
      "2020-02-12 16:00:02,401 Example #3\n",
      "2020-02-12 16:00:02,401 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:00:02,401 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:00:02,401 \tHypothesis: ( 1 Mbakorinte 4 : 1 ) Ka sha u tesen ér ka u eren kwagh u a lu a mi ga yô , ka u er kwagh u a lu a mi ga .\n",
      "2020-02-12 16:00:02,401 Validation result (greedy) at epoch   1, step     2000: bleu:   3.88, loss: 90648.3359, ppl:  21.3709, duration: 23.5734s\n",
      "2020-02-12 16:00:12,468 Epoch   1 Step:     2100 Batch Loss:     2.993847 Tokens per Sec:    22237, Lr: 0.000300\n",
      "2020-02-12 16:00:22,577 Epoch   1 Step:     2200 Batch Loss:     3.233315 Tokens per Sec:    22031, Lr: 0.000300\n",
      "2020-02-12 16:00:32,535 Epoch   1 Step:     2300 Batch Loss:     2.942458 Tokens per Sec:    22381, Lr: 0.000300\n",
      "2020-02-12 16:00:42,407 Epoch   1 Step:     2400 Batch Loss:     2.957521 Tokens per Sec:    22137, Lr: 0.000300\n",
      "2020-02-12 16:00:47,977 Epoch   1: total training loss 9082.40\n",
      "2020-02-12 16:00:47,978 EPOCH 2\n",
      "2020-02-12 16:00:52,569 Epoch   2 Step:     2500 Batch Loss:     2.771921 Tokens per Sec:    20169, Lr: 0.000300\n",
      "2020-02-12 16:01:02,396 Epoch   2 Step:     2600 Batch Loss:     2.767326 Tokens per Sec:    22158, Lr: 0.000300\n",
      "2020-02-12 16:01:12,241 Epoch   2 Step:     2700 Batch Loss:     2.821128 Tokens per Sec:    22720, Lr: 0.000300\n",
      "2020-02-12 16:01:22,095 Epoch   2 Step:     2800 Batch Loss:     2.964184 Tokens per Sec:    22927, Lr: 0.000300\n",
      "2020-02-12 16:01:31,914 Epoch   2 Step:     2900 Batch Loss:     3.064916 Tokens per Sec:    22691, Lr: 0.000300\n",
      "2020-02-12 16:01:41,772 Epoch   2 Step:     3000 Batch Loss:     2.732623 Tokens per Sec:    22483, Lr: 0.000300\n",
      "2020-02-12 16:02:06,481 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:02:06,481 Saving new checkpoint.\n",
      "2020-02-12 16:02:06,649 Example #0\n",
      "2020-02-12 16:02:06,649 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:02:06,649 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:02:06,650 \tHypothesis: Er i lu u civir Yehova a ishima i môm yô , ka u lun a aeren a dedoo a dedoo a dedoo a dedoo a a a lu a mi la , ka u tesen ér ka u eren kwagh u dedoo a lu a mi la , shi ka u tesen ér ka u eren kwagh u dedoo u dedoo u nan ve lu a mi la . — Pasalmi 119 : 10 - 11 .\n",
      "2020-02-12 16:02:06,650 Example #1\n",
      "2020-02-12 16:02:06,650 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:02:06,650 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:02:06,650 \tHypothesis: Se fatyô u kaan ser , “ Ka sha ci u se lu a mi la , man se lu a mi je la . ” — Ps .\n",
      "2020-02-12 16:02:06,650 Example #2\n",
      "2020-02-12 16:02:06,650 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:02:06,650 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:02:06,650 \tHypothesis: Nahan , Paulu lu ôron kwagh u i yer un ér Sefania la , a kaa ér : “ Ka sha ci u Mika .\n",
      "2020-02-12 16:02:06,650 Example #3\n",
      "2020-02-12 16:02:06,651 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:02:06,651 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:02:06,651 \tHypothesis: Ka kwagh u injaa u i gbe u se er kwagh u or u nan lu a mi la ga .\n",
      "2020-02-12 16:02:06,651 Validation result (greedy) at epoch   2, step     3000: bleu:   4.98, loss: 83058.2344, ppl:  16.5377, duration: 24.8784s\n",
      "2020-02-12 16:02:16,482 Epoch   2 Step:     3100 Batch Loss:     2.755773 Tokens per Sec:    22019, Lr: 0.000300\n",
      "2020-02-12 16:02:26,339 Epoch   2 Step:     3200 Batch Loss:     2.941496 Tokens per Sec:    22699, Lr: 0.000300\n",
      "2020-02-12 16:02:36,258 Epoch   2 Step:     3300 Batch Loss:     2.974449 Tokens per Sec:    22175, Lr: 0.000300\n",
      "2020-02-12 16:02:46,160 Epoch   2 Step:     3400 Batch Loss:     2.765259 Tokens per Sec:    21327, Lr: 0.000300\n",
      "2020-02-12 16:02:56,017 Epoch   2 Step:     3500 Batch Loss:     3.028700 Tokens per Sec:    22024, Lr: 0.000300\n",
      "2020-02-12 16:03:05,907 Epoch   2 Step:     3600 Batch Loss:     3.162649 Tokens per Sec:    23086, Lr: 0.000300\n",
      "2020-02-12 16:03:15,832 Epoch   2 Step:     3700 Batch Loss:     2.648297 Tokens per Sec:    22559, Lr: 0.000300\n",
      "2020-02-12 16:03:26,061 Epoch   2 Step:     3800 Batch Loss:     2.949837 Tokens per Sec:    21189, Lr: 0.000300\n",
      "2020-02-12 16:03:36,181 Epoch   2 Step:     3900 Batch Loss:     2.998596 Tokens per Sec:    21572, Lr: 0.000300\n",
      "2020-02-12 16:03:46,322 Epoch   2 Step:     4000 Batch Loss:     2.707163 Tokens per Sec:    22097, Lr: 0.000300\n",
      "2020-02-12 16:04:09,240 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:04:09,240 Saving new checkpoint.\n",
      "2020-02-12 16:04:09,409 Example #0\n",
      "2020-02-12 16:04:09,410 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:04:09,410 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:04:09,410 \tHypothesis: Er nan ve i gbe u or u nan lu a mi sha ci u Yehova yô , ka u nan lu a dooshima u nan a lu a mi la , ka u nan lu a mi sha u nan jighjigh u nan man u nan u nan man u nan u nan la . — Luka 6 : 9 - 9 .\n",
      "2020-02-12 16:04:09,410 Example #1\n",
      "2020-02-12 16:04:09,410 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:04:09,410 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:04:09,410 \tHypothesis: Se fatyô u kaan ser , “ Ka sha shighe u se lu a mi la , ” man se lu a mi sha shighe u se lu a mi la je ! — Ps .\n",
      "2020-02-12 16:04:09,410 Example #2\n",
      "2020-02-12 16:04:09,410 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:04:09,410 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:04:09,411 \tHypothesis: Ken masejime yô , Paulu lu anyom a karen imôngo , a lu ken Korinte , man Yohane man Yohane man Yohane .\n",
      "2020-02-12 16:04:09,411 Example #3\n",
      "2020-02-12 16:04:09,411 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:04:09,411 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:04:09,411 \tHypothesis: ( 1 Mbakorinte 3 : 1 ) Kwagh ne una fatyô u eren kwagh u a lu a mi ga .\n",
      "2020-02-12 16:04:09,411 Validation result (greedy) at epoch   2, step     4000: bleu:   7.26, loss: 77589.7812, ppl:  13.7484, duration: 23.0879s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:04:19,315 Epoch   2 Step:     4100 Batch Loss:     2.247452 Tokens per Sec:    22159, Lr: 0.000300\n",
      "2020-02-12 16:04:29,221 Epoch   2 Step:     4200 Batch Loss:     2.779771 Tokens per Sec:    22260, Lr: 0.000300\n",
      "2020-02-12 16:04:39,354 Epoch   2 Step:     4300 Batch Loss:     2.120739 Tokens per Sec:    21882, Lr: 0.000300\n",
      "2020-02-12 16:04:49,391 Epoch   2 Step:     4400 Batch Loss:     2.848859 Tokens per Sec:    21879, Lr: 0.000300\n",
      "2020-02-12 16:04:59,376 Epoch   2 Step:     4500 Batch Loss:     2.804246 Tokens per Sec:    22455, Lr: 0.000300\n",
      "2020-02-12 16:05:09,519 Epoch   2 Step:     4600 Batch Loss:     2.565914 Tokens per Sec:    21965, Lr: 0.000300\n",
      "2020-02-12 16:05:19,682 Epoch   2 Step:     4700 Batch Loss:     2.769480 Tokens per Sec:    21692, Lr: 0.000300\n",
      "2020-02-12 16:05:29,775 Epoch   2 Step:     4800 Batch Loss:     2.822880 Tokens per Sec:    21512, Lr: 0.000300\n",
      "2020-02-12 16:05:39,817 Epoch   2 Step:     4900 Batch Loss:     2.971874 Tokens per Sec:    21757, Lr: 0.000300\n",
      "2020-02-12 16:05:41,955 Epoch   2: total training loss 6929.75\n",
      "2020-02-12 16:05:41,955 EPOCH 3\n",
      "2020-02-12 16:05:50,184 Epoch   3 Step:     5000 Batch Loss:     2.465707 Tokens per Sec:    21611, Lr: 0.000300\n",
      "2020-02-12 16:06:10,123 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:06:10,123 Saving new checkpoint.\n",
      "2020-02-12 16:06:10,303 Example #0\n",
      "2020-02-12 16:06:10,303 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:06:10,303 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:06:10,304 \tHypothesis: Er nan ve i gbe u or u nan lu a inja i Yehova yô , ka u nan lu a inja i nan sha ci u nan lu a inja i nan ga , shin nan ngu a inja i nan sha dooshima u nan a lu a mi la . — Marku 6 : 9 - 9 .\n",
      "2020-02-12 16:06:10,304 Example #1\n",
      "2020-02-12 16:06:10,304 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:06:10,304 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:06:10,304 \tHypothesis: Se kpa se fatyô u nengen er “ ayange a masejime ” la , shi se fatyô u lun a inja kpishi ! — Mat .\n",
      "2020-02-12 16:06:10,304 Example #2\n",
      "2020-02-12 16:06:10,304 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:06:10,304 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:06:10,304 \tHypothesis: Ken anyom kar kar kar yô , Paulu lu ken mzough u ken Efese , u a lu ken Efese la .\n",
      "2020-02-12 16:06:10,304 Example #3\n",
      "2020-02-12 16:06:10,305 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:06:10,305 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:06:10,305 \tHypothesis: Kwagh ne una fatyô u lun a inja kpishi , nahan kwagh ne una fatyô u eren kwagh u a lu a mi ga .\n",
      "2020-02-12 16:06:10,305 Validation result (greedy) at epoch   3, step     5000: bleu:   9.31, loss: 73523.7891, ppl:  11.9840, duration: 20.1207s\n",
      "2020-02-12 16:06:20,256 Epoch   3 Step:     5100 Batch Loss:     2.387710 Tokens per Sec:    22109, Lr: 0.000300\n",
      "2020-02-12 16:06:30,107 Epoch   3 Step:     5200 Batch Loss:     2.558197 Tokens per Sec:    22568, Lr: 0.000300\n",
      "2020-02-12 16:06:39,985 Epoch   3 Step:     5300 Batch Loss:     2.618725 Tokens per Sec:    21998, Lr: 0.000300\n",
      "2020-02-12 16:06:49,981 Epoch   3 Step:     5400 Batch Loss:     2.448833 Tokens per Sec:    22080, Lr: 0.000300\n",
      "2020-02-12 16:06:59,848 Epoch   3 Step:     5500 Batch Loss:     2.614009 Tokens per Sec:    22340, Lr: 0.000300\n",
      "2020-02-12 16:07:09,770 Epoch   3 Step:     5600 Batch Loss:     2.469665 Tokens per Sec:    22276, Lr: 0.000300\n",
      "2020-02-12 16:07:19,707 Epoch   3 Step:     5700 Batch Loss:     2.449043 Tokens per Sec:    22438, Lr: 0.000300\n",
      "2020-02-12 16:07:29,688 Epoch   3 Step:     5800 Batch Loss:     2.619204 Tokens per Sec:    22450, Lr: 0.000300\n",
      "2020-02-12 16:07:39,714 Epoch   3 Step:     5900 Batch Loss:     2.670405 Tokens per Sec:    21551, Lr: 0.000300\n",
      "2020-02-12 16:07:49,805 Epoch   3 Step:     6000 Batch Loss:     2.711541 Tokens per Sec:    22025, Lr: 0.000300\n",
      "2020-02-12 16:08:11,891 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:08:11,891 Saving new checkpoint.\n",
      "2020-02-12 16:08:12,079 Example #0\n",
      "2020-02-12 16:08:12,079 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:08:12,080 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:08:12,080 \tHypothesis: Er Yehova a lu a isharen i eren ivese i vesen yô , ka u wasen mba ve lu ken mzough la , ka u wasen mba ve lu a dooshima la , shi ve lu a dooshima u a lu a mi sha u wasen ve u lun a dooshima la . — Marku 9 : 9 - 9 .\n",
      "2020-02-12 16:08:12,080 Example #1\n",
      "2020-02-12 16:08:12,080 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:08:12,080 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:08:12,080 \tHypothesis: Se kpa se fatyô u nengen “ shighe u se lu a mi ” la , shi se nenge a shighe u se lu a mi la . — Rom .\n",
      "2020-02-12 16:08:12,080 Example #2\n",
      "2020-02-12 16:08:12,080 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:08:12,080 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:08:12,080 \tHypothesis: Ken masejime yô , Paulu lu ken bem ken Efese , ken Efese , man lu ken Efese .\n",
      "2020-02-12 16:08:12,080 Example #3\n",
      "2020-02-12 16:08:12,081 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:08:12,081 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:08:12,081 \tHypothesis: ( 1 Mbakorinte 4 : 1 ) Kwagh ne una fatyô u eren kwagh u a lu u or u nan lu a mi la ga .\n",
      "2020-02-12 16:08:12,081 Validation result (greedy) at epoch   3, step     6000: bleu:  10.51, loss: 70898.3906, ppl:  10.9670, duration: 22.2757s\n",
      "2020-02-12 16:08:22,005 Epoch   3 Step:     6100 Batch Loss:     2.803612 Tokens per Sec:    21733, Lr: 0.000300\n",
      "2020-02-12 16:08:31,886 Epoch   3 Step:     6200 Batch Loss:     2.231687 Tokens per Sec:    22270, Lr: 0.000300\n",
      "2020-02-12 16:08:41,774 Epoch   3 Step:     6300 Batch Loss:     2.216193 Tokens per Sec:    22390, Lr: 0.000300\n",
      "2020-02-12 16:08:51,805 Epoch   3 Step:     6400 Batch Loss:     2.739138 Tokens per Sec:    22473, Lr: 0.000300\n",
      "2020-02-12 16:09:01,679 Epoch   3 Step:     6500 Batch Loss:     2.276480 Tokens per Sec:    22256, Lr: 0.000300\n",
      "2020-02-12 16:09:11,583 Epoch   3 Step:     6600 Batch Loss:     2.295534 Tokens per Sec:    22182, Lr: 0.000300\n",
      "2020-02-12 16:09:21,610 Epoch   3 Step:     6700 Batch Loss:     2.598059 Tokens per Sec:    22170, Lr: 0.000300\n",
      "2020-02-12 16:09:31,677 Epoch   3 Step:     6800 Batch Loss:     2.492252 Tokens per Sec:    21920, Lr: 0.000300\n",
      "2020-02-12 16:09:41,752 Epoch   3 Step:     6900 Batch Loss:     2.634245 Tokens per Sec:    22390, Lr: 0.000300\n",
      "2020-02-12 16:09:51,806 Epoch   3 Step:     7000 Batch Loss:     2.221498 Tokens per Sec:    21928, Lr: 0.000300\n",
      "2020-02-12 16:10:13,381 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:10:13,381 Saving new checkpoint.\n",
      "2020-02-12 16:10:13,567 Example #0\n",
      "2020-02-12 16:10:13,567 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:10:13,567 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:10:13,567 \tHypothesis: Er ivese i lu hange hange u Yehova a lu a mi sha u lun a inja yô , ka u lun a dooshima u a lu a mi sha u wasen mba ve lu ken dooshima la , shi ve lu a dooshima u ve lu a mi la kpaa . — Marku 9 : 9 - 9 .\n",
      "2020-02-12 16:10:13,568 Example #1\n",
      "2020-02-12 16:10:13,568 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:10:13,568 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:10:13,568 \tHypothesis: Se kpa se fatyô u nengen a “ shighe u i lu van ” la , shi se mba nengen a mi sha shighe u i lu van la . — Rom .\n",
      "2020-02-12 16:10:13,568 Example #2\n",
      "2020-02-12 16:10:13,568 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:10:13,568 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:10:13,568 \tHypothesis: Ken masejime yô , Paulu lu ken Efese , ken tar u Efese , shi lu ken Efese je .\n",
      "2020-02-12 16:10:13,568 Example #3\n",
      "2020-02-12 16:10:13,568 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:10:13,569 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:10:13,569 \tHypothesis: Kwagh ne una fatyô u lun or u nan lu a mi sha gbenda u nan lu a mi la ga .\n",
      "2020-02-12 16:10:13,569 Validation result (greedy) at epoch   3, step     7000: bleu:  11.87, loss: 68224.2266, ppl:  10.0198, duration: 21.7623s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:10:23,420 Epoch   3 Step:     7100 Batch Loss:     2.392289 Tokens per Sec:    22390, Lr: 0.000300\n",
      "2020-02-12 16:10:33,255 Epoch   3 Step:     7200 Batch Loss:     2.308585 Tokens per Sec:    22352, Lr: 0.000300\n",
      "2020-02-12 16:10:43,111 Epoch   3 Step:     7300 Batch Loss:     2.353991 Tokens per Sec:    22982, Lr: 0.000300\n",
      "2020-02-12 16:10:51,099 Epoch   3: total training loss 6184.23\n",
      "2020-02-12 16:10:51,099 EPOCH 4\n",
      "2020-02-12 16:10:53,407 Epoch   4 Step:     7400 Batch Loss:     2.594020 Tokens per Sec:    18604, Lr: 0.000300\n",
      "2020-02-12 16:11:03,264 Epoch   4 Step:     7500 Batch Loss:     2.548170 Tokens per Sec:    22402, Lr: 0.000300\n",
      "2020-02-12 16:11:13,196 Epoch   4 Step:     7600 Batch Loss:     2.366792 Tokens per Sec:    22935, Lr: 0.000300\n",
      "2020-02-12 16:11:23,085 Epoch   4 Step:     7700 Batch Loss:     2.471555 Tokens per Sec:    22248, Lr: 0.000300\n",
      "2020-02-12 16:11:32,991 Epoch   4 Step:     7800 Batch Loss:     2.077398 Tokens per Sec:    22014, Lr: 0.000300\n",
      "2020-02-12 16:11:42,916 Epoch   4 Step:     7900 Batch Loss:     2.559347 Tokens per Sec:    22381, Lr: 0.000300\n",
      "2020-02-12 16:11:52,902 Epoch   4 Step:     8000 Batch Loss:     2.120636 Tokens per Sec:    22261, Lr: 0.000300\n",
      "2020-02-12 16:12:12,928 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:12:12,928 Saving new checkpoint.\n",
      "2020-02-12 16:12:13,115 Example #0\n",
      "2020-02-12 16:12:13,115 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:12:13,115 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:12:13,115 \tHypothesis: Er ivese i vesen i Yehova a lu a mi sha ci u mba ve lu a mi la ka kwagh u hange hange yô , ka u lun a dooshima u a lu a mi la , ka u lun a dooshima man dooshima u ve a lu a mi ken dooshima la . — Marku 9 : 9 - 10 .\n",
      "2020-02-12 16:12:13,115 Example #1\n",
      "2020-02-12 16:12:13,115 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:12:13,116 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:12:13,116 \tHypothesis: Se kpa se fatyô u nengen “ shighe u mkur ” la , shi se mba a iwanger i kpilighyol kpen kpen ! — Rom .\n",
      "2020-02-12 16:12:13,116 Example #2\n",
      "2020-02-12 16:12:13,116 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:12:13,116 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:12:13,116 \tHypothesis: Ken anyom a karen ne , Paulu lu ken Efese , nahan a lu ken bem .\n",
      "2020-02-12 16:12:13,116 Example #3\n",
      "2020-02-12 16:12:13,116 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:12:13,116 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:12:13,116 \tHypothesis: Kwagh ne una fatyô u lun a inja sha gbenda u vesen u a lu a mi la ga .\n",
      "2020-02-12 16:12:13,116 Validation result (greedy) at epoch   4, step     8000: bleu:  13.13, loss: 66128.5703, ppl:   9.3350, duration: 20.2143s\n",
      "2020-02-12 16:12:23,104 Epoch   4 Step:     8100 Batch Loss:     2.234457 Tokens per Sec:    22093, Lr: 0.000300\n",
      "2020-02-12 16:12:33,030 Epoch   4 Step:     8200 Batch Loss:     2.701011 Tokens per Sec:    22751, Lr: 0.000300\n",
      "2020-02-12 16:12:42,909 Epoch   4 Step:     8300 Batch Loss:     2.481467 Tokens per Sec:    22401, Lr: 0.000300\n",
      "2020-02-12 16:12:52,825 Epoch   4 Step:     8400 Batch Loss:     2.432872 Tokens per Sec:    22500, Lr: 0.000300\n",
      "2020-02-12 16:13:02,840 Epoch   4 Step:     8500 Batch Loss:     2.528447 Tokens per Sec:    23059, Lr: 0.000300\n",
      "2020-02-12 16:13:12,701 Epoch   4 Step:     8600 Batch Loss:     2.325520 Tokens per Sec:    22765, Lr: 0.000300\n",
      "2020-02-12 16:13:22,552 Epoch   4 Step:     8700 Batch Loss:     2.364350 Tokens per Sec:    22233, Lr: 0.000300\n",
      "2020-02-12 16:13:32,427 Epoch   4 Step:     8800 Batch Loss:     2.461168 Tokens per Sec:    21916, Lr: 0.000300\n",
      "2020-02-12 16:13:42,554 Epoch   4 Step:     8900 Batch Loss:     2.226756 Tokens per Sec:    22067, Lr: 0.000300\n",
      "2020-02-12 16:13:52,682 Epoch   4 Step:     9000 Batch Loss:     2.398198 Tokens per Sec:    21802, Lr: 0.000300\n",
      "2020-02-12 16:14:14,582 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:14:14,582 Saving new checkpoint.\n",
      "2020-02-12 16:14:14,773 Example #0\n",
      "2020-02-12 16:14:14,773 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:14:14,773 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:14:14,773 \tHypothesis: Er ivese i lu sha ci u Yehova yô , ka hange hange u noov man kasev vea lu a dooshima , shin vea lu a dooshima u ve lu a mi la , shi vea lu a dooshima u ve lu a mi la kpaa . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:14:14,773 Example #1\n",
      "2020-02-12 16:14:14,773 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:14:14,773 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:14:14,774 \tHypothesis: Se kpa se fatyô u nengen “ shighe u doon ” ne , shi se mba a shighe u i lu van la . — Rom .\n",
      "2020-02-12 16:14:14,774 Example #2\n",
      "2020-02-12 16:14:14,774 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:14:14,774 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:14:14,774 \tHypothesis: Paulu yange er kwagh ne anyom kar imôngo ken tar u Efese , nahan a lu ken bem .\n",
      "2020-02-12 16:14:14,774 Example #3\n",
      "2020-02-12 16:14:14,774 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:14:14,774 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:14:14,774 \tHypothesis: Kwagh ne una fatyô u lun ormaren u nan lu a tahav mbu eren kwagh u nan soo la ga .\n",
      "2020-02-12 16:14:14,774 Validation result (greedy) at epoch   4, step     9000: bleu:  14.41, loss: 64338.4922, ppl:   8.7873, duration: 22.0916s\n",
      "2020-02-12 16:14:24,848 Epoch   4 Step:     9100 Batch Loss:     2.023343 Tokens per Sec:    22034, Lr: 0.000300\n",
      "2020-02-12 16:14:34,855 Epoch   4 Step:     9200 Batch Loss:     2.337622 Tokens per Sec:    22289, Lr: 0.000300\n",
      "2020-02-12 16:14:44,757 Epoch   4 Step:     9300 Batch Loss:     2.306293 Tokens per Sec:    22518, Lr: 0.000300\n",
      "2020-02-12 16:14:54,749 Epoch   4 Step:     9400 Batch Loss:     2.179697 Tokens per Sec:    22299, Lr: 0.000300\n",
      "2020-02-12 16:15:04,763 Epoch   4 Step:     9500 Batch Loss:     2.449395 Tokens per Sec:    22183, Lr: 0.000300\n",
      "2020-02-12 16:15:14,689 Epoch   4 Step:     9600 Batch Loss:     2.093209 Tokens per Sec:    21866, Lr: 0.000300\n",
      "2020-02-12 16:15:24,779 Epoch   4 Step:     9700 Batch Loss:     2.653842 Tokens per Sec:    22260, Lr: 0.000300\n",
      "2020-02-12 16:15:34,682 Epoch   4 Step:     9800 Batch Loss:     2.267682 Tokens per Sec:    21629, Lr: 0.000300\n",
      "2020-02-12 16:15:37,566 Epoch   4: total training loss 5722.38\n",
      "2020-02-12 16:15:37,566 EPOCH 5\n",
      "2020-02-12 16:15:44,904 Epoch   5 Step:     9900 Batch Loss:     1.966238 Tokens per Sec:    21832, Lr: 0.000300\n",
      "2020-02-12 16:15:54,844 Epoch   5 Step:    10000 Batch Loss:     2.487109 Tokens per Sec:    22467, Lr: 0.000300\n",
      "2020-02-12 16:16:14,759 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:16:14,759 Saving new checkpoint.\n",
      "2020-02-12 16:16:14,948 Example #0\n",
      "2020-02-12 16:16:14,948 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:16:14,949 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:16:14,949 \tHypothesis: Er ivese i lu i Yehova a lu a mi sha kwagh u ivesegh yô , ka hange hange u noov man kasev vea lu a dooshima u ve lu a mi la , shi vea lu a dooshima u ve lu a mi sha u eren kwagh u dooshima la kpaa . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:16:14,949 Example #1\n",
      "2020-02-12 16:16:14,949 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:16:14,949 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:16:14,949 \tHypothesis: Se kpa se fatyô u nengen “ shighe u doon ” ne , shi se mba a shighe u doon tsung ! — Rom .\n",
      "2020-02-12 16:16:14,949 Example #2\n",
      "2020-02-12 16:16:14,949 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:16:14,949 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:16:14,949 \tHypothesis: Ken anyom pue kar yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:16:14,950 Example #3\n",
      "2020-02-12 16:16:14,950 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:16:14,950 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:16:14,950 \tHypothesis: Kwagh ne una fatyô u eren kwagh u or u nan lu a mi sha gbenda u nan soo u eren la ga .\n",
      "2020-02-12 16:16:14,950 Validation result (greedy) at epoch   5, step    10000: bleu:  15.51, loss: 62378.2969, ppl:   8.2243, duration: 20.1053s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:16:24,819 Epoch   5 Step:    10100 Batch Loss:     1.982166 Tokens per Sec:    22377, Lr: 0.000300\n",
      "2020-02-12 16:16:34,695 Epoch   5 Step:    10200 Batch Loss:     2.273010 Tokens per Sec:    22155, Lr: 0.000300\n",
      "2020-02-12 16:16:44,558 Epoch   5 Step:    10300 Batch Loss:     2.021907 Tokens per Sec:    22818, Lr: 0.000300\n",
      "2020-02-12 16:16:54,489 Epoch   5 Step:    10400 Batch Loss:     2.374995 Tokens per Sec:    22376, Lr: 0.000300\n",
      "2020-02-12 16:17:04,457 Epoch   5 Step:    10500 Batch Loss:     2.002748 Tokens per Sec:    22170, Lr: 0.000300\n",
      "2020-02-12 16:17:14,423 Epoch   5 Step:    10600 Batch Loss:     2.368401 Tokens per Sec:    21915, Lr: 0.000300\n",
      "2020-02-12 16:17:24,329 Epoch   5 Step:    10700 Batch Loss:     2.217776 Tokens per Sec:    22016, Lr: 0.000300\n",
      "2020-02-12 16:17:34,280 Epoch   5 Step:    10800 Batch Loss:     1.833345 Tokens per Sec:    22474, Lr: 0.000300\n",
      "2020-02-12 16:17:44,366 Epoch   5 Step:    10900 Batch Loss:     2.128143 Tokens per Sec:    21638, Lr: 0.000300\n",
      "2020-02-12 16:17:54,498 Epoch   5 Step:    11000 Batch Loss:     2.304907 Tokens per Sec:    22062, Lr: 0.000300\n",
      "2020-02-12 16:18:15,938 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:18:15,939 Saving new checkpoint.\n",
      "2020-02-12 16:18:16,126 Example #0\n",
      "2020-02-12 16:18:16,127 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:18:16,127 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:18:16,127 \tHypothesis: Er Yehova a lu a ian i eren ivese i vesen yô , ka hange hange u noov man kasev vea lu a dooshima shi vea lu a dooshima u ve lu a mi la , vea lu a dooshima u ve lu a mi la , vea lu a mi sha u lun a dooshima u Aôndo . — Marku 6 : 9 - 9 .\n",
      "2020-02-12 16:18:16,127 Example #1\n",
      "2020-02-12 16:18:16,127 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:18:16,127 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:18:16,127 \tHypothesis: Se kpa se fatyô u nengen a “ shighe u kpilighyol ” — Rom .\n",
      "2020-02-12 16:18:16,127 Example #2\n",
      "2020-02-12 16:18:16,128 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:18:16,128 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:18:16,128 \tHypothesis: Ken anyom pue kar yô , Paulu lu ken Masedonia man Timoteu .\n",
      "2020-02-12 16:18:16,128 Example #3\n",
      "2020-02-12 16:18:16,128 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:18:16,128 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:18:16,128 \tHypothesis: Alaghga or nana fatyô u lun a imba gbenda la sha gbenda u nan soo u eren kwagh u nan soo la ga .\n",
      "2020-02-12 16:18:16,128 Validation result (greedy) at epoch   5, step    11000: bleu:  15.66, loss: 61080.1367, ppl:   7.8714, duration: 21.6300s\n",
      "2020-02-12 16:18:26,323 Epoch   5 Step:    11100 Batch Loss:     2.487456 Tokens per Sec:    21972, Lr: 0.000300\n",
      "2020-02-12 16:18:36,370 Epoch   5 Step:    11200 Batch Loss:     2.282746 Tokens per Sec:    22104, Lr: 0.000300\n",
      "2020-02-12 16:18:46,378 Epoch   5 Step:    11300 Batch Loss:     2.207912 Tokens per Sec:    21580, Lr: 0.000300\n",
      "2020-02-12 16:18:56,373 Epoch   5 Step:    11400 Batch Loss:     2.507225 Tokens per Sec:    21892, Lr: 0.000300\n",
      "2020-02-12 16:19:06,311 Epoch   5 Step:    11500 Batch Loss:     2.211894 Tokens per Sec:    22288, Lr: 0.000300\n",
      "2020-02-12 16:19:16,292 Epoch   5 Step:    11600 Batch Loss:     2.177078 Tokens per Sec:    22076, Lr: 0.000300\n",
      "2020-02-12 16:19:26,169 Epoch   5 Step:    11700 Batch Loss:     2.290599 Tokens per Sec:    22705, Lr: 0.000300\n",
      "2020-02-12 16:19:36,062 Epoch   5 Step:    11800 Batch Loss:     2.441172 Tokens per Sec:    22458, Lr: 0.000300\n",
      "2020-02-12 16:19:45,980 Epoch   5 Step:    11900 Batch Loss:     1.976811 Tokens per Sec:    21963, Lr: 0.000300\n",
      "2020-02-12 16:19:55,868 Epoch   5 Step:    12000 Batch Loss:     2.123380 Tokens per Sec:    22256, Lr: 0.000300\n",
      "2020-02-12 16:20:15,868 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:20:15,868 Saving new checkpoint.\n",
      "2020-02-12 16:20:16,057 Example #0\n",
      "2020-02-12 16:20:16,058 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:20:16,058 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:20:16,058 \tHypothesis: Er ivese i lu i Yehova a lu a mi la i lu kwagh u vesen u vesen yô , ka hange hange u noov man kasev vea lu a dooshima u ve lu a mi la , vea lu a mi sha u lun a dooshima man dooshima u ve lu a mi la . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:20:16,058 Example #1\n",
      "2020-02-12 16:20:16,058 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:20:16,058 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:20:16,058 \tHypothesis: Se kpa se fatyô u ‘ nengen a shighe u doon tsung , ’ shi se mba a shighe u doon tsung je kpaa ! — Rom .\n",
      "2020-02-12 16:20:16,058 Example #2\n",
      "2020-02-12 16:20:16,059 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:20:16,059 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:20:16,059 \tHypothesis: Shighe kar yô , Paulu lu ken Masedonia man Timoteu , ken Efese .\n",
      "2020-02-12 16:20:16,059 Example #3\n",
      "2020-02-12 16:20:16,059 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:20:16,059 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:20:16,059 \tHypothesis: Kwagh ne una fatyô u lun ormaren u lun a inja sha gbenda u i gbe u nana er kwagh sha mi ga yô .\n",
      "2020-02-12 16:20:16,059 Validation result (greedy) at epoch   5, step    12000: bleu:  16.63, loss: 59583.0781, ppl:   7.4833, duration: 20.1909s\n",
      "2020-02-12 16:20:25,904 Epoch   5 Step:    12100 Batch Loss:     2.391531 Tokens per Sec:    22426, Lr: 0.000300\n",
      "2020-02-12 16:20:35,758 Epoch   5 Step:    12200 Batch Loss:     2.188322 Tokens per Sec:    22498, Lr: 0.000300\n",
      "2020-02-12 16:20:44,605 Epoch   5: total training loss 5428.09\n",
      "2020-02-12 16:20:44,605 EPOCH 6\n",
      "2020-02-12 16:20:45,882 Epoch   6 Step:    12300 Batch Loss:     2.308112 Tokens per Sec:    17816, Lr: 0.000300\n",
      "2020-02-12 16:20:55,822 Epoch   6 Step:    12400 Batch Loss:     1.998952 Tokens per Sec:    22357, Lr: 0.000300\n",
      "2020-02-12 16:21:05,708 Epoch   6 Step:    12500 Batch Loss:     2.275053 Tokens per Sec:    22686, Lr: 0.000300\n",
      "2020-02-12 16:21:15,612 Epoch   6 Step:    12600 Batch Loss:     2.252143 Tokens per Sec:    22486, Lr: 0.000300\n",
      "2020-02-12 16:21:25,643 Epoch   6 Step:    12700 Batch Loss:     2.169776 Tokens per Sec:    22205, Lr: 0.000300\n",
      "2020-02-12 16:21:35,512 Epoch   6 Step:    12800 Batch Loss:     1.907391 Tokens per Sec:    21538, Lr: 0.000300\n",
      "2020-02-12 16:21:45,430 Epoch   6 Step:    12900 Batch Loss:     2.397996 Tokens per Sec:    22504, Lr: 0.000300\n",
      "2020-02-12 16:21:55,386 Epoch   6 Step:    13000 Batch Loss:     2.280136 Tokens per Sec:    22440, Lr: 0.000300\n",
      "2020-02-12 16:22:14,869 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:22:14,869 Saving new checkpoint.\n",
      "2020-02-12 16:22:15,058 Example #0\n",
      "2020-02-12 16:22:15,059 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:22:15,059 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:22:15,059 \tHypothesis: Er Yehova a lu a awashima u vough yô , ka hange hange u noov man kasev vea lu a isharen i eren kwagh u dedoo , shi vea lu a dooshima u ve lu a mi la , vea lu a mi sha u tesen dooshima u ve lu a mi la . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:22:15,059 Example #1\n",
      "2020-02-12 16:22:15,059 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:22:15,059 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:22:15,059 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , shi se mba ikyua a Yehova kpaa . — Rom .\n",
      "2020-02-12 16:22:15,059 Example #2\n",
      "2020-02-12 16:22:15,060 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:22:15,060 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:22:15,060 \tHypothesis: Ken anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:22:15,060 Example #3\n",
      "2020-02-12 16:22:15,060 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:22:15,060 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:22:15,060 \tHypothesis: Alaghga or nana fatyô u eren kwagh u nan soo u eren la sha gbenda u nan lu a mi ga .\n",
      "2020-02-12 16:22:15,060 Validation result (greedy) at epoch   6, step    13000: bleu:  16.84, loss: 58631.8945, ppl:   7.2467, duration: 19.6735s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:22:25,010 Epoch   6 Step:    13100 Batch Loss:     2.109484 Tokens per Sec:    22544, Lr: 0.000300\n",
      "2020-02-12 16:22:34,934 Epoch   6 Step:    13200 Batch Loss:     1.922879 Tokens per Sec:    21784, Lr: 0.000300\n",
      "2020-02-12 16:22:44,877 Epoch   6 Step:    13300 Batch Loss:     2.186911 Tokens per Sec:    22893, Lr: 0.000300\n",
      "2020-02-12 16:22:54,871 Epoch   6 Step:    13400 Batch Loss:     1.936656 Tokens per Sec:    21614, Lr: 0.000300\n",
      "2020-02-12 16:23:04,832 Epoch   6 Step:    13500 Batch Loss:     2.064526 Tokens per Sec:    22143, Lr: 0.000300\n",
      "2020-02-12 16:23:14,760 Epoch   6 Step:    13600 Batch Loss:     2.143647 Tokens per Sec:    22323, Lr: 0.000300\n",
      "2020-02-12 16:23:24,762 Epoch   6 Step:    13700 Batch Loss:     2.014436 Tokens per Sec:    22092, Lr: 0.000300\n",
      "2020-02-12 16:23:34,673 Epoch   6 Step:    13800 Batch Loss:     1.766890 Tokens per Sec:    22426, Lr: 0.000300\n",
      "2020-02-12 16:23:44,490 Epoch   6 Step:    13900 Batch Loss:     1.993079 Tokens per Sec:    21993, Lr: 0.000300\n",
      "2020-02-12 16:23:54,419 Epoch   6 Step:    14000 Batch Loss:     1.970243 Tokens per Sec:    22196, Lr: 0.000300\n",
      "2020-02-12 16:24:13,490 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:24:13,491 Saving new checkpoint.\n",
      "2020-02-12 16:24:13,690 Example #0\n",
      "2020-02-12 16:24:13,691 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:24:13,691 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:24:13,691 \tHypothesis: Er i lu kwagh u ivesegh i Yehova a lu a mi sha u lun uma gbem yô , ka hange hange u noov man kasev vea lu a dooshima u ve lu a mi la , shi vea lu a mi sha dooshima u ve lu a mi la kpaa . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:24:13,691 Example #1\n",
      "2020-02-12 16:24:13,691 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:24:13,691 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:24:13,691 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , shi se mba ikyua a vese kpaa ! — Rom .\n",
      "2020-02-12 16:24:13,691 Example #2\n",
      "2020-02-12 16:24:13,691 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:24:13,691 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:24:13,692 \tHypothesis: Ken masejime yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:24:13,692 Example #3\n",
      "2020-02-12 16:24:13,692 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:24:13,692 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:24:13,692 \tHypothesis: Kwagh ne una fatyô u lun or u nan lu a mi sha gbenda u i lu u nan or u nan lu a mi la ga .\n",
      "2020-02-12 16:24:13,692 Validation result (greedy) at epoch   6, step    14000: bleu:  17.81, loss: 57596.1562, ppl:   6.9975, duration: 19.2720s\n",
      "2020-02-12 16:24:23,728 Epoch   6 Step:    14100 Batch Loss:     2.200467 Tokens per Sec:    22066, Lr: 0.000300\n",
      "2020-02-12 16:24:33,678 Epoch   6 Step:    14200 Batch Loss:     2.461322 Tokens per Sec:    22432, Lr: 0.000300\n",
      "2020-02-12 16:24:43,582 Epoch   6 Step:    14300 Batch Loss:     2.316701 Tokens per Sec:    22904, Lr: 0.000300\n",
      "2020-02-12 16:24:53,516 Epoch   6 Step:    14400 Batch Loss:     1.990537 Tokens per Sec:    22172, Lr: 0.000300\n",
      "2020-02-12 16:25:03,426 Epoch   6 Step:    14500 Batch Loss:     2.032058 Tokens per Sec:    22148, Lr: 0.000300\n",
      "2020-02-12 16:25:13,430 Epoch   6 Step:    14600 Batch Loss:     1.834708 Tokens per Sec:    22052, Lr: 0.000300\n",
      "2020-02-12 16:25:23,378 Epoch   6 Step:    14700 Batch Loss:     2.131001 Tokens per Sec:    21988, Lr: 0.000300\n",
      "2020-02-12 16:25:28,152 Epoch   6: total training loss 5174.43\n",
      "2020-02-12 16:25:28,153 EPOCH 7\n",
      "2020-02-12 16:25:33,686 Epoch   7 Step:    14800 Batch Loss:     2.117682 Tokens per Sec:    21193, Lr: 0.000300\n",
      "2020-02-12 16:25:43,579 Epoch   7 Step:    14900 Batch Loss:     2.088548 Tokens per Sec:    21896, Lr: 0.000300\n",
      "2020-02-12 16:25:53,570 Epoch   7 Step:    15000 Batch Loss:     2.110936 Tokens per Sec:    21967, Lr: 0.000300\n",
      "2020-02-12 16:26:13,398 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:26:13,398 Saving new checkpoint.\n",
      "2020-02-12 16:26:13,588 Example #0\n",
      "2020-02-12 16:26:13,588 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:26:13,588 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:26:13,588 \tHypothesis: Er ivese i lu kwagh u vesen u Yehova a lu a mi yô , ka hange hange u noov man kasev vea nôngo kwagh kpoghuloo sha er vea lu a dooshima man dooshima u ve lu a mi la , shi vea seer lun a dooshima u a lu ken mzough la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:26:13,588 Example #1\n",
      "2020-02-12 16:26:13,588 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:26:13,589 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:26:13,589 \tHypothesis: Se kpa se fatyô u nengen “ shighe ” ne , nahan ka shighe u kpilighyol je la ! — Rom .\n",
      "2020-02-12 16:26:13,589 Example #2\n",
      "2020-02-12 16:26:13,589 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:26:13,589 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:26:13,589 \tHypothesis: Ken anyom nga kar hegen yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:26:13,589 Example #3\n",
      "2020-02-12 16:26:13,589 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:26:13,589 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:26:13,589 \tHypothesis: A fatyô u kaan ér or u nan lu a tahav mbu eren kwagh sha inja ga yô .\n",
      "2020-02-12 16:26:13,589 Validation result (greedy) at epoch   7, step    15000: bleu:  18.55, loss: 56781.5078, ppl:   6.8076, duration: 20.0184s\n",
      "2020-02-12 16:26:23,593 Epoch   7 Step:    15100 Batch Loss:     2.274362 Tokens per Sec:    22315, Lr: 0.000300\n",
      "2020-02-12 16:26:33,578 Epoch   7 Step:    15200 Batch Loss:     2.229963 Tokens per Sec:    22084, Lr: 0.000300\n",
      "2020-02-12 16:26:43,528 Epoch   7 Step:    15300 Batch Loss:     1.852999 Tokens per Sec:    21807, Lr: 0.000300\n",
      "2020-02-12 16:26:53,515 Epoch   7 Step:    15400 Batch Loss:     1.877712 Tokens per Sec:    22103, Lr: 0.000300\n",
      "2020-02-12 16:27:03,424 Epoch   7 Step:    15500 Batch Loss:     2.152976 Tokens per Sec:    22229, Lr: 0.000300\n",
      "2020-02-12 16:27:13,506 Epoch   7 Step:    15600 Batch Loss:     2.379538 Tokens per Sec:    21676, Lr: 0.000300\n",
      "2020-02-12 16:27:23,701 Epoch   7 Step:    15700 Batch Loss:     1.854567 Tokens per Sec:    21686, Lr: 0.000300\n",
      "2020-02-12 16:27:33,916 Epoch   7 Step:    15800 Batch Loss:     1.512889 Tokens per Sec:    21490, Lr: 0.000300\n",
      "2020-02-12 16:27:44,029 Epoch   7 Step:    15900 Batch Loss:     2.030560 Tokens per Sec:    21496, Lr: 0.000300\n",
      "2020-02-12 16:27:54,053 Epoch   7 Step:    16000 Batch Loss:     1.885270 Tokens per Sec:    22011, Lr: 0.000300\n",
      "2020-02-12 16:28:13,204 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:28:13,204 Saving new checkpoint.\n",
      "2020-02-12 16:28:13,391 Example #0\n",
      "2020-02-12 16:28:13,392 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:28:13,392 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:28:13,392 \tHypothesis: Er i lu hange hange u nom man kwase vea lu a dooshima yô , ka hange hange u vea nôngo kwagh kpoghuloo vea za hemen u lun a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u ve lu a mi sha kwagh u dooshima la . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:28:13,392 Example #1\n",
      "2020-02-12 16:28:13,392 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:28:13,392 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:28:13,392 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , shi ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:28:13,392 Example #2\n",
      "2020-02-12 16:28:13,393 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:28:13,393 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:28:13,393 \tHypothesis: Ken anyom pue kar yô , Paulu lu ken Masedonia man Timoteu .\n",
      "2020-02-12 16:28:13,393 Example #3\n",
      "2020-02-12 16:28:13,393 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:28:13,393 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:28:13,393 \tHypothesis: Kwagh ne una fatyô u lun nahan sha gbenda u i gbe u or a er kwagh sha mi ga la .\n",
      "2020-02-12 16:28:13,393 Validation result (greedy) at epoch   7, step    16000: bleu:  18.55, loss: 55835.7148, ppl:   6.5935, duration: 19.3396s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:28:23,312 Epoch   7 Step:    16100 Batch Loss:     2.202749 Tokens per Sec:    22046, Lr: 0.000300\n",
      "2020-02-12 16:28:33,240 Epoch   7 Step:    16200 Batch Loss:     1.774728 Tokens per Sec:    22458, Lr: 0.000300\n",
      "2020-02-12 16:28:43,167 Epoch   7 Step:    16300 Batch Loss:     1.727866 Tokens per Sec:    22257, Lr: 0.000300\n",
      "2020-02-12 16:28:53,155 Epoch   7 Step:    16400 Batch Loss:     1.883956 Tokens per Sec:    22405, Lr: 0.000300\n",
      "2020-02-12 16:29:03,116 Epoch   7 Step:    16500 Batch Loss:     1.944116 Tokens per Sec:    22219, Lr: 0.000300\n",
      "2020-02-12 16:29:13,100 Epoch   7 Step:    16600 Batch Loss:     2.092608 Tokens per Sec:    21733, Lr: 0.000300\n",
      "2020-02-12 16:29:23,135 Epoch   7 Step:    16700 Batch Loss:     2.157441 Tokens per Sec:    21804, Lr: 0.000300\n",
      "2020-02-12 16:29:33,063 Epoch   7 Step:    16800 Batch Loss:     2.321883 Tokens per Sec:    22356, Lr: 0.000300\n",
      "2020-02-12 16:29:43,059 Epoch   7 Step:    16900 Batch Loss:     2.048239 Tokens per Sec:    22476, Lr: 0.000300\n",
      "2020-02-12 16:29:53,051 Epoch   7 Step:    17000 Batch Loss:     2.074275 Tokens per Sec:    22484, Lr: 0.000300\n",
      "2020-02-12 16:30:13,380 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:30:13,380 Saving new checkpoint.\n",
      "2020-02-12 16:30:13,579 Example #0\n",
      "2020-02-12 16:30:13,579 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:30:13,579 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:30:13,579 \tHypothesis: Er Yehova a ver ivese sha u kuren awashima na u tsôron yô , gba hange hange u noov vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi la , vea za hemen u lun a dooshima u a lu ken mzough la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:30:13,580 Example #1\n",
      "2020-02-12 16:30:13,580 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:30:13,580 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:30:13,580 \tHypothesis: Se kpa se fatyô u nengen “ shighe ” u doon tsung ne , man ka shighe u doon je la . — Rom .\n",
      "2020-02-12 16:30:13,580 Example #2\n",
      "2020-02-12 16:30:13,580 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:30:13,580 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:30:13,580 \tHypothesis: Ken masejime yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:30:13,580 Example #3\n",
      "2020-02-12 16:30:13,580 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:30:13,581 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:30:13,581 \tHypothesis: Alaghga or nana fatyô u eren kwagh u a lu u shami ga yô .\n",
      "2020-02-12 16:30:13,581 Validation result (greedy) at epoch   7, step    17000: bleu:  18.45, loss: 55276.6172, ppl:   6.4702, duration: 20.5290s\n",
      "2020-02-12 16:30:23,623 Epoch   7 Step:    17100 Batch Loss:     2.066772 Tokens per Sec:    22174, Lr: 0.000300\n",
      "2020-02-12 16:30:33,695 Epoch   7 Step:    17200 Batch Loss:     2.134908 Tokens per Sec:    22273, Lr: 0.000300\n",
      "2020-02-12 16:30:34,706 Epoch   7: total training loss 5007.08\n",
      "2020-02-12 16:30:34,707 EPOCH 8\n",
      "2020-02-12 16:30:44,048 Epoch   8 Step:    17300 Batch Loss:     1.992836 Tokens per Sec:    21293, Lr: 0.000300\n",
      "2020-02-12 16:30:54,245 Epoch   8 Step:    17400 Batch Loss:     2.089068 Tokens per Sec:    21298, Lr: 0.000300\n",
      "2020-02-12 16:31:04,383 Epoch   8 Step:    17500 Batch Loss:     1.974792 Tokens per Sec:    21779, Lr: 0.000300\n",
      "2020-02-12 16:31:14,499 Epoch   8 Step:    17600 Batch Loss:     1.596745 Tokens per Sec:    21819, Lr: 0.000300\n",
      "2020-02-12 16:31:24,619 Epoch   8 Step:    17700 Batch Loss:     1.985769 Tokens per Sec:    21598, Lr: 0.000300\n",
      "2020-02-12 16:31:34,727 Epoch   8 Step:    17800 Batch Loss:     2.368596 Tokens per Sec:    21567, Lr: 0.000300\n",
      "2020-02-12 16:31:44,962 Epoch   8 Step:    17900 Batch Loss:     2.099503 Tokens per Sec:    21588, Lr: 0.000300\n",
      "2020-02-12 16:31:55,057 Epoch   8 Step:    18000 Batch Loss:     2.024249 Tokens per Sec:    21395, Lr: 0.000300\n",
      "2020-02-12 16:32:14,614 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:32:14,614 Saving new checkpoint.\n",
      "2020-02-12 16:32:14,801 Example #0\n",
      "2020-02-12 16:32:14,801 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:32:14,801 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:32:14,802 \tHypothesis: Er Yehova a lu a ian i eren kwagh u a tsough u eren la yô , gba hange hange u noov vea nôngo kwagh kpoghuloo ve vea lu a ieren i dedoo i vea lu a mi i lun a dooshima man dooshima u ve lu a mi la vea za hemen u lun a mi ken dooshima u a lu ken mzough la . — Marku 10 : 9 - 9 .\n",
      "2020-02-12 16:32:14,802 Example #1\n",
      "2020-02-12 16:32:14,802 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:32:14,802 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:32:14,802 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 16:32:14,802 Example #2\n",
      "2020-02-12 16:32:14,802 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:32:14,802 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:32:14,802 \tHypothesis: Ken anyom nga kar kpuaa yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:32:14,802 Example #3\n",
      "2020-02-12 16:32:14,803 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:32:14,803 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:32:14,803 \tHypothesis: Alaghga or u nan lu a tahav mbu eren kwagh sha gbenda u nan lu a mi ga yô .\n",
      "2020-02-12 16:32:14,803 Validation result (greedy) at epoch   8, step    18000: bleu:  19.18, loss: 54626.7266, ppl:   6.3297, duration: 19.7450s\n",
      "2020-02-12 16:32:24,744 Epoch   8 Step:    18100 Batch Loss:     2.188540 Tokens per Sec:    22123, Lr: 0.000300\n",
      "2020-02-12 16:32:34,623 Epoch   8 Step:    18200 Batch Loss:     1.939353 Tokens per Sec:    23081, Lr: 0.000300\n",
      "2020-02-12 16:32:44,478 Epoch   8 Step:    18300 Batch Loss:     2.117628 Tokens per Sec:    21803, Lr: 0.000300\n",
      "2020-02-12 16:32:54,455 Epoch   8 Step:    18400 Batch Loss:     2.267230 Tokens per Sec:    21838, Lr: 0.000300\n",
      "2020-02-12 16:33:04,427 Epoch   8 Step:    18500 Batch Loss:     2.009772 Tokens per Sec:    22636, Lr: 0.000300\n",
      "2020-02-12 16:33:14,384 Epoch   8 Step:    18600 Batch Loss:     2.147149 Tokens per Sec:    22323, Lr: 0.000300\n",
      "2020-02-12 16:33:24,405 Epoch   8 Step:    18700 Batch Loss:     1.758848 Tokens per Sec:    22389, Lr: 0.000300\n",
      "2020-02-12 16:33:34,278 Epoch   8 Step:    18800 Batch Loss:     2.362626 Tokens per Sec:    22279, Lr: 0.000300\n",
      "2020-02-12 16:33:44,210 Epoch   8 Step:    18900 Batch Loss:     1.595763 Tokens per Sec:    22522, Lr: 0.000300\n",
      "2020-02-12 16:33:54,230 Epoch   8 Step:    19000 Batch Loss:     2.155347 Tokens per Sec:    21937, Lr: 0.000300\n",
      "2020-02-12 16:34:14,117 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:34:14,117 Saving new checkpoint.\n",
      "2020-02-12 16:34:14,816 Example #0\n",
      "2020-02-12 16:34:14,816 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:34:14,816 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:34:14,816 \tHypothesis: Er i lu u nom man kwase vea er ivese ve i lu sha inja yô , ka hange hange u vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u ve lu a mi la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:34:14,816 Example #1\n",
      "2020-02-12 16:34:14,816 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:34:14,817 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:34:14,817 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 16:34:14,817 Example #2\n",
      "2020-02-12 16:34:14,817 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:34:14,817 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:34:14,817 \tHypothesis: Ken anyom pue la , Paulu lu ken Masedonia man Timoteu .\n",
      "2020-02-12 16:34:14,817 Example #3\n",
      "2020-02-12 16:34:14,817 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:34:14,817 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:34:14,817 \tHypothesis: Alaghga or u nan lu a tahav mbu eren kwagh u a lu u shami ga la nana lu a mi ga .\n",
      "2020-02-12 16:34:14,817 Validation result (greedy) at epoch   8, step    19000: bleu:  19.30, loss: 53792.4180, ppl:   6.1538, duration: 20.5869s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:34:24,755 Epoch   8 Step:    19100 Batch Loss:     1.704111 Tokens per Sec:    22431, Lr: 0.000300\n",
      "2020-02-12 16:34:34,710 Epoch   8 Step:    19200 Batch Loss:     2.043770 Tokens per Sec:    22612, Lr: 0.000300\n",
      "2020-02-12 16:34:44,838 Epoch   8 Step:    19300 Batch Loss:     1.879913 Tokens per Sec:    22152, Lr: 0.000300\n",
      "2020-02-12 16:34:55,046 Epoch   8 Step:    19400 Batch Loss:     2.035709 Tokens per Sec:    21450, Lr: 0.000300\n",
      "2020-02-12 16:35:05,213 Epoch   8 Step:    19500 Batch Loss:     1.800448 Tokens per Sec:    21968, Lr: 0.000300\n",
      "2020-02-12 16:35:15,358 Epoch   8 Step:    19600 Batch Loss:     1.916712 Tokens per Sec:    22098, Lr: 0.000300\n",
      "2020-02-12 16:35:22,135 Epoch   8: total training loss 4850.83\n",
      "2020-02-12 16:35:22,135 EPOCH 9\n",
      "2020-02-12 16:35:25,547 Epoch   9 Step:    19700 Batch Loss:     1.914348 Tokens per Sec:    20597, Lr: 0.000300\n",
      "2020-02-12 16:35:35,406 Epoch   9 Step:    19800 Batch Loss:     2.093338 Tokens per Sec:    22488, Lr: 0.000300\n",
      "2020-02-12 16:35:45,319 Epoch   9 Step:    19900 Batch Loss:     1.731291 Tokens per Sec:    21581, Lr: 0.000300\n",
      "2020-02-12 16:35:55,409 Epoch   9 Step:    20000 Batch Loss:     1.859691 Tokens per Sec:    21975, Lr: 0.000300\n",
      "2020-02-12 16:36:13,974 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:36:13,974 Saving new checkpoint.\n",
      "2020-02-12 16:36:14,163 Example #0\n",
      "2020-02-12 16:36:14,163 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:36:14,163 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:36:14,164 \tHypothesis: Er Yehova a lu ivesegh yô , ka hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u a lu ken dooshima la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:36:14,164 Example #1\n",
      "2020-02-12 16:36:14,164 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:36:14,164 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:36:14,164 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:36:14,164 Example #2\n",
      "2020-02-12 16:36:14,164 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:36:14,164 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:36:14,164 \tHypothesis: Ken anyom a kar hegen ne , Paulu lu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 16:36:14,164 Example #3\n",
      "2020-02-12 16:36:14,165 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:36:14,165 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:36:14,165 \tHypothesis: ( Mateu 24 : 45 ) A fatyô u kaan ér or ka nana er kwagh sha gbenda u nan soo la ga .\n",
      "2020-02-12 16:36:14,165 Validation result (greedy) at epoch   9, step    20000: bleu:  19.93, loss: 53332.3789, ppl:   6.0589, duration: 18.7555s\n",
      "2020-02-12 16:36:24,263 Epoch   9 Step:    20100 Batch Loss:     1.730392 Tokens per Sec:    21967, Lr: 0.000300\n",
      "2020-02-12 16:36:34,395 Epoch   9 Step:    20200 Batch Loss:     2.260367 Tokens per Sec:    21366, Lr: 0.000300\n",
      "2020-02-12 16:36:44,530 Epoch   9 Step:    20300 Batch Loss:     2.002793 Tokens per Sec:    21622, Lr: 0.000300\n",
      "2020-02-12 16:36:54,741 Epoch   9 Step:    20400 Batch Loss:     2.105446 Tokens per Sec:    21319, Lr: 0.000300\n",
      "2020-02-12 16:37:04,804 Epoch   9 Step:    20500 Batch Loss:     1.825804 Tokens per Sec:    22186, Lr: 0.000300\n",
      "2020-02-12 16:37:14,690 Epoch   9 Step:    20600 Batch Loss:     1.893557 Tokens per Sec:    21966, Lr: 0.000300\n",
      "2020-02-12 16:37:24,580 Epoch   9 Step:    20700 Batch Loss:     2.006706 Tokens per Sec:    22943, Lr: 0.000300\n",
      "2020-02-12 16:37:34,430 Epoch   9 Step:    20800 Batch Loss:     1.526448 Tokens per Sec:    21744, Lr: 0.000300\n",
      "2020-02-12 16:37:44,321 Epoch   9 Step:    20900 Batch Loss:     1.718555 Tokens per Sec:    22066, Lr: 0.000300\n",
      "2020-02-12 16:37:54,335 Epoch   9 Step:    21000 Batch Loss:     2.056952 Tokens per Sec:    22429, Lr: 0.000300\n",
      "2020-02-12 16:38:13,847 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:38:13,847 Saving new checkpoint.\n",
      "2020-02-12 16:38:14,036 Example #0\n",
      "2020-02-12 16:38:14,037 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:38:14,037 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:38:14,037 \tHypothesis: Er ivese i lu kwagh u vesen u Yehova a ver la i lu kwagh u vesen yô , ka hange hange u noov man kasev vea nôngo kwagh kpoghuloo ve vea za hemen u lun a dooshima u ve lu a mi sha kwagh u dooshima u ve lu a mi ken tar u dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:38:14,037 Example #1\n",
      "2020-02-12 16:38:14,037 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:38:14,037 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:38:14,037 \tHypothesis: Se kpa se fatyô u nengen “ shighe ” ne , nahan se mba ikyua a na ! — Rom .\n",
      "2020-02-12 16:38:14,037 Example #2\n",
      "2020-02-12 16:38:14,037 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:38:14,037 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:38:14,037 \tHypothesis: Ken anyom pue la , Paulu lu ken Masedonia man Timoteu .\n",
      "2020-02-12 16:38:14,038 Example #3\n",
      "2020-02-12 16:38:14,038 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:38:14,038 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:38:14,038 \tHypothesis: Alaghga a lu u or a er kwagh sha gbenda u vough ga .\n",
      "2020-02-12 16:38:14,038 Validation result (greedy) at epoch   9, step    21000: bleu:  20.49, loss: 52872.0273, ppl:   5.9654, duration: 19.7021s\n",
      "2020-02-12 16:38:23,941 Epoch   9 Step:    21100 Batch Loss:     1.799833 Tokens per Sec:    22347, Lr: 0.000300\n",
      "2020-02-12 16:38:33,825 Epoch   9 Step:    21200 Batch Loss:     1.947842 Tokens per Sec:    22760, Lr: 0.000300\n",
      "2020-02-12 16:38:43,708 Epoch   9 Step:    21300 Batch Loss:     1.995403 Tokens per Sec:    22389, Lr: 0.000300\n",
      "2020-02-12 16:38:53,719 Epoch   9 Step:    21400 Batch Loss:     2.032055 Tokens per Sec:    21861, Lr: 0.000300\n",
      "2020-02-12 16:39:03,643 Epoch   9 Step:    21500 Batch Loss:     1.759507 Tokens per Sec:    21907, Lr: 0.000300\n",
      "2020-02-12 16:39:13,660 Epoch   9 Step:    21600 Batch Loss:     1.925145 Tokens per Sec:    21727, Lr: 0.000300\n",
      "2020-02-12 16:39:23,796 Epoch   9 Step:    21700 Batch Loss:     2.001884 Tokens per Sec:    22486, Lr: 0.000300\n",
      "2020-02-12 16:39:33,635 Epoch   9 Step:    21800 Batch Loss:     1.871985 Tokens per Sec:    22036, Lr: 0.000300\n",
      "2020-02-12 16:39:43,447 Epoch   9 Step:    21900 Batch Loss:     2.024700 Tokens per Sec:    22562, Lr: 0.000300\n",
      "2020-02-12 16:39:53,336 Epoch   9 Step:    22000 Batch Loss:     2.112658 Tokens per Sec:    22287, Lr: 0.000300\n",
      "2020-02-12 16:40:13,212 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:40:13,212 Saving new checkpoint.\n",
      "2020-02-12 16:40:13,410 Example #0\n",
      "2020-02-12 16:40:13,411 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:40:13,411 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:40:13,411 \tHypothesis: Er ivese i lu kwagh u vesen u Yehova a lu a mi yô , gba hange hange u noov man kasev vea nôngo kwagh kpoghuloo sha er vea za hemen u lun a dooshima shi vea za hemen u lun a dooshima u a lu hen atô ve la yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:40:13,411 Example #1\n",
      "2020-02-12 16:40:13,411 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:40:13,411 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:40:13,411 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 16:40:13,411 Example #2\n",
      "2020-02-12 16:40:13,411 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:40:13,412 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:40:13,412 \tHypothesis: Ken anyom pue la , Paulu lu ken gar u Masedonia man Timoteu .\n",
      "2020-02-12 16:40:13,412 Example #3\n",
      "2020-02-12 16:40:13,412 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:40:13,412 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:40:13,412 \tHypothesis: Alaghga or u nan lu a ian i eren kwagh sha gbenda u a lu u vough ga yô .\n",
      "2020-02-12 16:40:13,412 Validation result (greedy) at epoch   9, step    22000: bleu:  20.59, loss: 52241.1719, ppl:   5.8396, duration: 20.0752s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:40:23,256 Epoch   9 Step:    22100 Batch Loss:     1.694650 Tokens per Sec:    22721, Lr: 0.000300\n",
      "2020-02-12 16:40:26,694 Epoch   9: total training loss 4742.62\n",
      "2020-02-12 16:40:26,694 EPOCH 10\n",
      "2020-02-12 16:40:33,391 Epoch  10 Step:    22200 Batch Loss:     1.824729 Tokens per Sec:    22062, Lr: 0.000300\n",
      "2020-02-12 16:40:43,318 Epoch  10 Step:    22300 Batch Loss:     1.830457 Tokens per Sec:    22359, Lr: 0.000300\n",
      "2020-02-12 16:40:53,228 Epoch  10 Step:    22400 Batch Loss:     1.765829 Tokens per Sec:    21978, Lr: 0.000300\n",
      "2020-02-12 16:41:03,132 Epoch  10 Step:    22500 Batch Loss:     1.860943 Tokens per Sec:    22552, Lr: 0.000300\n",
      "2020-02-12 16:41:13,053 Epoch  10 Step:    22600 Batch Loss:     1.998243 Tokens per Sec:    22675, Lr: 0.000300\n",
      "2020-02-12 16:41:22,940 Epoch  10 Step:    22700 Batch Loss:     1.764805 Tokens per Sec:    22413, Lr: 0.000300\n",
      "2020-02-12 16:41:32,848 Epoch  10 Step:    22800 Batch Loss:     2.115628 Tokens per Sec:    22279, Lr: 0.000300\n",
      "2020-02-12 16:41:42,965 Epoch  10 Step:    22900 Batch Loss:     1.914941 Tokens per Sec:    21676, Lr: 0.000300\n",
      "2020-02-12 16:41:53,071 Epoch  10 Step:    23000 Batch Loss:     2.006256 Tokens per Sec:    21552, Lr: 0.000300\n",
      "2020-02-12 16:42:12,284 Example #0\n",
      "2020-02-12 16:42:12,284 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:42:12,284 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:42:12,285 \tHypothesis: Er Yehova a lu ivesegh i ivesegh yô , ka hange hange u noov man kasev vea nôngo sha afatyô ve cii sha er vea lu a dooshima shi vea seer lun a dooshima u a lu ken dooshima u a lu hen atô ve la yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:42:12,285 Example #1\n",
      "2020-02-12 16:42:12,285 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:42:12,285 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:42:12,285 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:42:12,285 Example #2\n",
      "2020-02-12 16:42:12,285 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:42:12,285 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:42:12,285 \tHypothesis: Ken anyom pue la , Paulu lu ken gar u Masedonia man Timoteu .\n",
      "2020-02-12 16:42:12,285 Example #3\n",
      "2020-02-12 16:42:12,285 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:42:12,286 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:42:12,286 \tHypothesis: Alaghga or u nan lu a ishima i môm la nana fatyô u lun sha gbenda u vough ga .\n",
      "2020-02-12 16:42:12,286 Validation result (greedy) at epoch  10, step    23000: bleu:  20.23, loss: 52303.1328, ppl:   5.8519, duration: 19.2143s\n",
      "2020-02-12 16:42:22,306 Epoch  10 Step:    23100 Batch Loss:     1.815794 Tokens per Sec:    22101, Lr: 0.000300\n",
      "2020-02-12 16:42:32,260 Epoch  10 Step:    23200 Batch Loss:     1.794646 Tokens per Sec:    21541, Lr: 0.000300\n",
      "2020-02-12 16:42:42,129 Epoch  10 Step:    23300 Batch Loss:     1.811051 Tokens per Sec:    22865, Lr: 0.000300\n",
      "2020-02-12 16:42:52,016 Epoch  10 Step:    23400 Batch Loss:     1.560463 Tokens per Sec:    22004, Lr: 0.000300\n",
      "2020-02-12 16:43:01,867 Epoch  10 Step:    23500 Batch Loss:     1.826227 Tokens per Sec:    22784, Lr: 0.000300\n",
      "2020-02-12 16:43:11,728 Epoch  10 Step:    23600 Batch Loss:     2.103531 Tokens per Sec:    22769, Lr: 0.000300\n",
      "2020-02-12 16:43:21,549 Epoch  10 Step:    23700 Batch Loss:     1.933853 Tokens per Sec:    22266, Lr: 0.000300\n",
      "2020-02-12 16:43:31,390 Epoch  10 Step:    23800 Batch Loss:     2.052123 Tokens per Sec:    23129, Lr: 0.000300\n",
      "2020-02-12 16:43:41,265 Epoch  10 Step:    23900 Batch Loss:     1.910568 Tokens per Sec:    22354, Lr: 0.000300\n",
      "2020-02-12 16:43:51,226 Epoch  10 Step:    24000 Batch Loss:     1.875479 Tokens per Sec:    21846, Lr: 0.000300\n",
      "2020-02-12 16:44:08,702 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:44:08,702 Saving new checkpoint.\n",
      "2020-02-12 16:44:08,891 Example #0\n",
      "2020-02-12 16:44:08,891 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:44:08,891 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:44:08,891 \tHypothesis: Er ivese i Yehova i lu kwagh môm a mlu u vough yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha kwagh u dooshima la , vea seer lun a dooshima u a lu ken dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:44:08,891 Example #1\n",
      "2020-02-12 16:44:08,892 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:44:08,892 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:44:08,892 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 16:44:08,892 Example #2\n",
      "2020-02-12 16:44:08,892 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:44:08,892 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:44:08,892 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu .\n",
      "2020-02-12 16:44:08,892 Example #3\n",
      "2020-02-12 16:44:08,892 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:44:08,892 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:44:08,893 \tHypothesis: Ka kwagh u or a er ve nana er ve nana lu a inja ga yô .\n",
      "2020-02-12 16:44:08,893 Validation result (greedy) at epoch  10, step    24000: bleu:  19.93, loss: 51739.8398, ppl:   5.7416, duration: 17.6659s\n",
      "2020-02-12 16:44:18,862 Epoch  10 Step:    24100 Batch Loss:     1.889015 Tokens per Sec:    22427, Lr: 0.000300\n",
      "2020-02-12 16:44:28,745 Epoch  10 Step:    24200 Batch Loss:     1.684302 Tokens per Sec:    21780, Lr: 0.000300\n",
      "2020-02-12 16:44:38,660 Epoch  10 Step:    24300 Batch Loss:     1.914559 Tokens per Sec:    22128, Lr: 0.000300\n",
      "2020-02-12 16:44:48,617 Epoch  10 Step:    24400 Batch Loss:     2.252124 Tokens per Sec:    21814, Lr: 0.000300\n",
      "2020-02-12 16:44:58,517 Epoch  10 Step:    24500 Batch Loss:     1.879817 Tokens per Sec:    22261, Lr: 0.000300\n",
      "2020-02-12 16:45:07,861 Epoch  10: total training loss 4624.46\n",
      "2020-02-12 16:45:07,861 EPOCH 11\n",
      "2020-02-12 16:45:08,738 Epoch  11 Step:    24600 Batch Loss:     1.897348 Tokens per Sec:    12198, Lr: 0.000300\n",
      "2020-02-12 16:45:18,668 Epoch  11 Step:    24700 Batch Loss:     1.911436 Tokens per Sec:    22010, Lr: 0.000300\n",
      "2020-02-12 16:45:28,669 Epoch  11 Step:    24800 Batch Loss:     2.049130 Tokens per Sec:    22006, Lr: 0.000300\n",
      "2020-02-12 16:45:38,715 Epoch  11 Step:    24900 Batch Loss:     1.639738 Tokens per Sec:    21548, Lr: 0.000300\n",
      "2020-02-12 16:45:48,876 Epoch  11 Step:    25000 Batch Loss:     1.771585 Tokens per Sec:    21238, Lr: 0.000300\n",
      "2020-02-12 16:46:07,700 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:46:07,700 Saving new checkpoint.\n",
      "2020-02-12 16:46:07,900 Example #0\n",
      "2020-02-12 16:46:07,900 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:46:07,900 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:46:07,900 \tHypothesis: Er Yehova a ver ivese i lun gbem sha won yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi sha dooshima la shi vea lu a dooshima u mimi sha kwagh u dooshima u ve lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:46:07,900 Example #1\n",
      "2020-02-12 16:46:07,900 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:46:07,900 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:46:07,900 \tHypothesis: Se kpa se fatyô u “ nengen ” imba shighe u kpilighyol ne , man ka mgbôghom je la . — Rom .\n",
      "2020-02-12 16:46:07,900 Example #2\n",
      "2020-02-12 16:46:07,901 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:46:07,901 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:46:07,901 \tHypothesis: Ken anyom pue a karen ne , Paulu ngu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 16:46:07,901 Example #3\n",
      "2020-02-12 16:46:07,901 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:46:07,901 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:46:07,901 \tHypothesis: Alaghga a lu u or nana lu a ishima i nan sha gbenda u i doo u nana er kwagh sha mi ga yô .\n",
      "2020-02-12 16:46:07,901 Validation result (greedy) at epoch  11, step    25000: bleu:  21.48, loss: 51244.4805, ppl:   5.6463, duration: 19.0251s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:46:17,886 Epoch  11 Step:    25100 Batch Loss:     1.992092 Tokens per Sec:    22669, Lr: 0.000300\n",
      "2020-02-12 16:46:27,791 Epoch  11 Step:    25200 Batch Loss:     1.829686 Tokens per Sec:    22128, Lr: 0.000300\n",
      "2020-02-12 16:46:37,644 Epoch  11 Step:    25300 Batch Loss:     1.771660 Tokens per Sec:    22004, Lr: 0.000300\n",
      "2020-02-12 16:46:47,580 Epoch  11 Step:    25400 Batch Loss:     1.828283 Tokens per Sec:    22475, Lr: 0.000300\n",
      "2020-02-12 16:46:57,458 Epoch  11 Step:    25500 Batch Loss:     1.573067 Tokens per Sec:    21617, Lr: 0.000300\n",
      "2020-02-12 16:47:07,500 Epoch  11 Step:    25600 Batch Loss:     1.939762 Tokens per Sec:    21885, Lr: 0.000300\n",
      "2020-02-12 16:47:17,446 Epoch  11 Step:    25700 Batch Loss:     1.876702 Tokens per Sec:    22402, Lr: 0.000300\n",
      "2020-02-12 16:47:27,453 Epoch  11 Step:    25800 Batch Loss:     1.681799 Tokens per Sec:    22376, Lr: 0.000300\n",
      "2020-02-12 16:47:37,336 Epoch  11 Step:    25900 Batch Loss:     1.932091 Tokens per Sec:    22418, Lr: 0.000300\n",
      "2020-02-12 16:47:47,256 Epoch  11 Step:    26000 Batch Loss:     1.547129 Tokens per Sec:    21657, Lr: 0.000300\n",
      "2020-02-12 16:48:06,386 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:48:06,387 Saving new checkpoint.\n",
      "2020-02-12 16:48:06,575 Example #0\n",
      "2020-02-12 16:48:06,575 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:48:06,575 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:48:06,575 \tHypothesis: Er i lu hange hange u nom man kwase vea lu a dooshima u mimi yô , ka hange hange u vea nôngo sha er vea za hemen u lun a dooshima u ve lu a mi sha u tesen dooshima la , shi vea za hemen u lun a dooshima u a lu ken dooshima la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:48:06,575 Example #1\n",
      "2020-02-12 16:48:06,575 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:48:06,575 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:48:06,576 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka kwagh u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:48:06,576 Example #2\n",
      "2020-02-12 16:48:06,576 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:48:06,576 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:48:06,576 \tHypothesis: Anyom nga kar pue yô , Paulu ngu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 16:48:06,576 Example #3\n",
      "2020-02-12 16:48:06,576 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:48:06,576 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:48:06,576 \tHypothesis: A fatyô u kaan ér or u nan lu a inja yô , ka u nan eren kwagh sha inja ga .\n",
      "2020-02-12 16:48:06,576 Validation result (greedy) at epoch  11, step    26000: bleu:  21.42, loss: 51034.2227, ppl:   5.6063, duration: 19.3196s\n",
      "2020-02-12 16:48:16,467 Epoch  11 Step:    26100 Batch Loss:     1.776893 Tokens per Sec:    22423, Lr: 0.000300\n",
      "2020-02-12 16:48:26,742 Epoch  11 Step:    26200 Batch Loss:     1.725854 Tokens per Sec:    21500, Lr: 0.000300\n",
      "2020-02-12 16:48:36,827 Epoch  11 Step:    26300 Batch Loss:     1.515929 Tokens per Sec:    22276, Lr: 0.000300\n",
      "2020-02-12 16:48:46,883 Epoch  11 Step:    26400 Batch Loss:     1.957026 Tokens per Sec:    22299, Lr: 0.000300\n",
      "2020-02-12 16:48:56,881 Epoch  11 Step:    26500 Batch Loss:     2.141359 Tokens per Sec:    22566, Lr: 0.000300\n",
      "2020-02-12 16:49:06,967 Epoch  11 Step:    26600 Batch Loss:     1.896598 Tokens per Sec:    21993, Lr: 0.000300\n",
      "2020-02-12 16:49:17,102 Epoch  11 Step:    26700 Batch Loss:     1.614436 Tokens per Sec:    22177, Lr: 0.000300\n",
      "2020-02-12 16:49:27,239 Epoch  11 Step:    26800 Batch Loss:     1.972823 Tokens per Sec:    21934, Lr: 0.000300\n",
      "2020-02-12 16:49:37,203 Epoch  11 Step:    26900 Batch Loss:     1.526728 Tokens per Sec:    22500, Lr: 0.000300\n",
      "2020-02-12 16:49:47,155 Epoch  11 Step:    27000 Batch Loss:     2.176866 Tokens per Sec:    21909, Lr: 0.000300\n",
      "2020-02-12 16:50:05,627 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:50:05,627 Saving new checkpoint.\n",
      "2020-02-12 16:50:05,816 Example #0\n",
      "2020-02-12 16:50:05,817 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:50:05,817 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:50:05,817 \tHypothesis: Er ivese i Yehova i lu kwagh u vesen u i gbe u a er yô , ka hange hange u noov man kasev vea nôngo kpoghuloo vea za hemen u lun a dooshima u ve lu a mi sha kwagh u dooshima la , shi vea za hemen u lun a mi ken dooshima u a lu hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:50:05,817 Example #1\n",
      "2020-02-12 16:50:05,817 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:50:05,817 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:50:05,817 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:50:05,817 Example #2\n",
      "2020-02-12 16:50:05,817 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:50:05,817 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:50:05,817 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 16:50:05,818 Example #3\n",
      "2020-02-12 16:50:05,818 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:50:05,818 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:50:05,818 \tHypothesis: Alaghga a lu u or nana er kwagh sha gbenda u i doo u nana er kwagh sha mi ga yô .\n",
      "2020-02-12 16:50:05,818 Validation result (greedy) at epoch  11, step    27000: bleu:  21.24, loss: 50560.1797, ppl:   5.5173, duration: 18.6618s\n",
      "2020-02-12 16:50:11,248 Epoch  11: total training loss 4543.82\n",
      "2020-02-12 16:50:11,248 EPOCH 12\n",
      "2020-02-12 16:50:15,984 Epoch  12 Step:    27100 Batch Loss:     1.761245 Tokens per Sec:    21246, Lr: 0.000300\n",
      "2020-02-12 16:50:25,994 Epoch  12 Step:    27200 Batch Loss:     1.672341 Tokens per Sec:    22088, Lr: 0.000300\n",
      "2020-02-12 16:50:35,994 Epoch  12 Step:    27300 Batch Loss:     1.925917 Tokens per Sec:    22314, Lr: 0.000300\n",
      "2020-02-12 16:50:45,928 Epoch  12 Step:    27400 Batch Loss:     2.294695 Tokens per Sec:    22220, Lr: 0.000300\n",
      "2020-02-12 16:50:55,831 Epoch  12 Step:    27500 Batch Loss:     1.935423 Tokens per Sec:    22098, Lr: 0.000300\n",
      "2020-02-12 16:51:05,743 Epoch  12 Step:    27600 Batch Loss:     2.015524 Tokens per Sec:    21841, Lr: 0.000300\n",
      "2020-02-12 16:51:15,640 Epoch  12 Step:    27700 Batch Loss:     1.969784 Tokens per Sec:    22834, Lr: 0.000300\n",
      "2020-02-12 16:51:25,638 Epoch  12 Step:    27800 Batch Loss:     1.635526 Tokens per Sec:    22183, Lr: 0.000300\n",
      "2020-02-12 16:51:35,580 Epoch  12 Step:    27900 Batch Loss:     1.618926 Tokens per Sec:    22099, Lr: 0.000300\n",
      "2020-02-12 16:51:45,485 Epoch  12 Step:    28000 Batch Loss:     1.790464 Tokens per Sec:    21960, Lr: 0.000300\n",
      "2020-02-12 16:52:05,194 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:52:05,194 Saving new checkpoint.\n",
      "2020-02-12 16:52:05,386 Example #0\n",
      "2020-02-12 16:52:05,387 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:52:05,387 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:52:05,387 \tHypothesis: Er i gbe u nom man kwase vea er ivese ve ia lu gbem sha won yô , doo u vea nôngo sha afatyô ve cii vea kura dooshima ve shi vea za hemen u lun ken dooshima u ve lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:52:05,387 Example #1\n",
      "2020-02-12 16:52:05,387 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:52:05,387 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:52:05,387 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:52:05,387 Example #2\n",
      "2020-02-12 16:52:05,387 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:52:05,388 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:52:05,388 \tHypothesis: Ken anyom pue la , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:52:05,388 Example #3\n",
      "2020-02-12 16:52:05,388 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:52:05,388 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:52:05,388 \tHypothesis: Ka u or nana fatyô u eren kwagh sha gbenda u i doo u nana er ga yô .\n",
      "2020-02-12 16:52:05,388 Validation result (greedy) at epoch  12, step    28000: bleu:  22.26, loss: 50354.1367, ppl:   5.4790, duration: 19.9025s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:52:15,423 Epoch  12 Step:    28100 Batch Loss:     1.539151 Tokens per Sec:    22240, Lr: 0.000300\n",
      "2020-02-12 16:52:25,321 Epoch  12 Step:    28200 Batch Loss:     2.161488 Tokens per Sec:    22817, Lr: 0.000300\n",
      "2020-02-12 16:52:35,323 Epoch  12 Step:    28300 Batch Loss:     1.646714 Tokens per Sec:    22131, Lr: 0.000300\n",
      "2020-02-12 16:52:45,142 Epoch  12 Step:    28400 Batch Loss:     1.863586 Tokens per Sec:    22631, Lr: 0.000300\n",
      "2020-02-12 16:52:55,061 Epoch  12 Step:    28500 Batch Loss:     1.740035 Tokens per Sec:    22251, Lr: 0.000300\n",
      "2020-02-12 16:53:05,019 Epoch  12 Step:    28600 Batch Loss:     1.642973 Tokens per Sec:    22294, Lr: 0.000300\n",
      "2020-02-12 16:53:15,136 Epoch  12 Step:    28700 Batch Loss:     1.612443 Tokens per Sec:    21869, Lr: 0.000300\n",
      "2020-02-12 16:53:25,262 Epoch  12 Step:    28800 Batch Loss:     1.866432 Tokens per Sec:    21660, Lr: 0.000300\n",
      "2020-02-12 16:53:35,381 Epoch  12 Step:    28900 Batch Loss:     1.835360 Tokens per Sec:    21702, Lr: 0.000300\n",
      "2020-02-12 16:53:45,488 Epoch  12 Step:    29000 Batch Loss:     1.639821 Tokens per Sec:    22025, Lr: 0.000300\n",
      "2020-02-12 16:54:04,513 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:54:04,514 Saving new checkpoint.\n",
      "2020-02-12 16:54:04,704 Example #0\n",
      "2020-02-12 16:54:04,705 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:54:04,705 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:54:04,705 \tHypothesis: Er ivaa i lu kwagh u vesen u Yehova a ver la i lu kwagh u vesen yô , gba u noov man kasev vea nôngo sha afatyô ve cii vea kura dooshima u ve lu a mi sha kwagh u dooshima la shi vea za hemen u lun a mi ken dooshima u a lu hen atô ve la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:54:04,705 Example #1\n",
      "2020-02-12 16:54:04,705 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:54:04,705 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:54:04,705 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:54:04,705 Example #2\n",
      "2020-02-12 16:54:04,705 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:54:04,706 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:54:04,706 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:54:04,706 Example #3\n",
      "2020-02-12 16:54:04,706 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:54:04,706 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:54:04,706 \tHypothesis: Alaghga or u nan lu a ian i eren kwagh u vough ga la nana lu a mi sha gbenda u vough ga .\n",
      "2020-02-12 16:54:04,706 Validation result (greedy) at epoch  12, step    29000: bleu:  21.92, loss: 49823.4375, ppl:   5.3817, duration: 19.2179s\n",
      "2020-02-12 16:54:14,821 Epoch  12 Step:    29100 Batch Loss:     1.782477 Tokens per Sec:    21795, Lr: 0.000300\n",
      "2020-02-12 16:54:25,031 Epoch  12 Step:    29200 Batch Loss:     1.604911 Tokens per Sec:    21501, Lr: 0.000300\n",
      "2020-02-12 16:54:35,242 Epoch  12 Step:    29300 Batch Loss:     1.641777 Tokens per Sec:    21687, Lr: 0.000300\n",
      "2020-02-12 16:54:45,437 Epoch  12 Step:    29400 Batch Loss:     1.831326 Tokens per Sec:    21611, Lr: 0.000300\n",
      "2020-02-12 16:54:55,577 Epoch  12 Step:    29500 Batch Loss:     2.090666 Tokens per Sec:    21907, Lr: 0.000300\n",
      "2020-02-12 16:54:56,888 Epoch  12: total training loss 4455.50\n",
      "2020-02-12 16:54:56,888 EPOCH 13\n",
      "2020-02-12 16:55:05,994 Epoch  13 Step:    29600 Batch Loss:     1.682260 Tokens per Sec:    20918, Lr: 0.000300\n",
      "2020-02-12 16:55:16,075 Epoch  13 Step:    29700 Batch Loss:     1.913036 Tokens per Sec:    21744, Lr: 0.000300\n",
      "2020-02-12 16:55:26,163 Epoch  13 Step:    29800 Batch Loss:     2.056014 Tokens per Sec:    22100, Lr: 0.000300\n",
      "2020-02-12 16:55:36,158 Epoch  13 Step:    29900 Batch Loss:     1.697894 Tokens per Sec:    22117, Lr: 0.000300\n",
      "2020-02-12 16:55:46,096 Epoch  13 Step:    30000 Batch Loss:     1.921973 Tokens per Sec:    22366, Lr: 0.000300\n",
      "2020-02-12 16:56:05,226 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:56:05,227 Saving new checkpoint.\n",
      "2020-02-12 16:56:05,424 Example #0\n",
      "2020-02-12 16:56:05,424 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:56:05,425 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:56:05,425 \tHypothesis: Er ivese i lu kwagh môm u Yehova a ver sha u kuren awashima na u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u a lu hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:56:05,425 Example #1\n",
      "2020-02-12 16:56:05,425 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:56:05,425 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:56:05,425 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:56:05,425 Example #2\n",
      "2020-02-12 16:56:05,425 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:56:05,425 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:56:05,425 \tHypothesis: Ken masejime yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 16:56:05,425 Example #3\n",
      "2020-02-12 16:56:05,426 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:56:05,426 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:56:05,426 \tHypothesis: Ka u or nana fatyô u eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 16:56:05,426 Validation result (greedy) at epoch  13, step    30000: bleu:  22.66, loss: 49705.7539, ppl:   5.3603, duration: 19.3291s\n",
      "2020-02-12 16:56:15,382 Epoch  13 Step:    30100 Batch Loss:     2.022979 Tokens per Sec:    22106, Lr: 0.000300\n",
      "2020-02-12 16:56:25,292 Epoch  13 Step:    30200 Batch Loss:     2.112054 Tokens per Sec:    22570, Lr: 0.000300\n",
      "2020-02-12 16:56:35,226 Epoch  13 Step:    30300 Batch Loss:     1.920949 Tokens per Sec:    22938, Lr: 0.000300\n",
      "2020-02-12 16:56:45,222 Epoch  13 Step:    30400 Batch Loss:     2.101107 Tokens per Sec:    21670, Lr: 0.000300\n",
      "2020-02-12 16:56:55,510 Epoch  13 Step:    30500 Batch Loss:     1.791649 Tokens per Sec:    21174, Lr: 0.000300\n",
      "2020-02-12 16:57:05,892 Epoch  13 Step:    30600 Batch Loss:     1.932489 Tokens per Sec:    21307, Lr: 0.000300\n",
      "2020-02-12 16:57:16,122 Epoch  13 Step:    30700 Batch Loss:     1.753716 Tokens per Sec:    21766, Lr: 0.000300\n",
      "2020-02-12 16:57:26,376 Epoch  13 Step:    30800 Batch Loss:     1.856784 Tokens per Sec:    21614, Lr: 0.000300\n",
      "2020-02-12 16:57:36,572 Epoch  13 Step:    30900 Batch Loss:     1.683047 Tokens per Sec:    21210, Lr: 0.000300\n",
      "2020-02-12 16:57:46,697 Epoch  13 Step:    31000 Batch Loss:     1.871389 Tokens per Sec:    21758, Lr: 0.000300\n",
      "2020-02-12 16:58:05,318 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 16:58:05,318 Saving new checkpoint.\n",
      "2020-02-12 16:58:05,512 Example #0\n",
      "2020-02-12 16:58:05,512 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 16:58:05,512 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 16:58:05,512 \tHypothesis: Er ivese i Yehova i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 16:58:05,513 Example #1\n",
      "2020-02-12 16:58:05,513 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 16:58:05,513 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 16:58:05,513 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 16:58:05,513 Example #2\n",
      "2020-02-12 16:58:05,513 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 16:58:05,513 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 16:58:05,513 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 16:58:05,513 Example #3\n",
      "2020-02-12 16:58:05,514 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 16:58:05,514 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 16:58:05,514 \tHypothesis: Alaghga or nana fatyô u eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 16:58:05,514 Validation result (greedy) at epoch  13, step    31000: bleu:  21.84, loss: 49343.9414, ppl:   5.2952, duration: 18.8164s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 16:58:15,541 Epoch  13 Step:    31100 Batch Loss:     1.357395 Tokens per Sec:    22289, Lr: 0.000300\n",
      "2020-02-12 16:58:25,467 Epoch  13 Step:    31200 Batch Loss:     1.795151 Tokens per Sec:    22656, Lr: 0.000300\n",
      "2020-02-12 16:58:35,370 Epoch  13 Step:    31300 Batch Loss:     1.561645 Tokens per Sec:    22346, Lr: 0.000300\n",
      "2020-02-12 16:58:45,341 Epoch  13 Step:    31400 Batch Loss:     1.726090 Tokens per Sec:    22120, Lr: 0.000300\n",
      "2020-02-12 16:58:55,484 Epoch  13 Step:    31500 Batch Loss:     1.642398 Tokens per Sec:    21676, Lr: 0.000300\n",
      "2020-02-12 16:59:05,659 Epoch  13 Step:    31600 Batch Loss:     1.837268 Tokens per Sec:    21398, Lr: 0.000300\n",
      "2020-02-12 16:59:15,825 Epoch  13 Step:    31700 Batch Loss:     2.092063 Tokens per Sec:    22196, Lr: 0.000300\n",
      "2020-02-12 16:59:25,894 Epoch  13 Step:    31800 Batch Loss:     1.542736 Tokens per Sec:    21513, Lr: 0.000300\n",
      "2020-02-12 16:59:35,842 Epoch  13 Step:    31900 Batch Loss:     1.935450 Tokens per Sec:    22193, Lr: 0.000300\n",
      "2020-02-12 16:59:43,126 Epoch  13: total training loss 4387.53\n",
      "2020-02-12 16:59:43,126 EPOCH 14\n",
      "2020-02-12 16:59:46,126 Epoch  14 Step:    32000 Batch Loss:     2.012460 Tokens per Sec:    19688, Lr: 0.000300\n",
      "2020-02-12 17:00:05,014 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:00:05,015 Saving new checkpoint.\n",
      "2020-02-12 17:00:05,215 Example #0\n",
      "2020-02-12 17:00:05,215 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:00:05,215 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:00:05,215 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea za hemen u lun a dooshima u mimi , shi vea za hemen u lun ken dooshima u a lu hen atô ve la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:00:05,215 Example #1\n",
      "2020-02-12 17:00:05,215 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:00:05,216 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:00:05,216 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 17:00:05,216 Example #2\n",
      "2020-02-12 17:00:05,216 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:00:05,216 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:00:05,216 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:00:05,216 Example #3\n",
      "2020-02-12 17:00:05,216 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:00:05,216 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:00:05,216 \tHypothesis: Alaghga or a lu a ian i eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:00:05,216 Validation result (greedy) at epoch  14, step    32000: bleu:  22.68, loss: 49154.0352, ppl:   5.2613, duration: 19.0894s\n",
      "2020-02-12 17:00:15,166 Epoch  14 Step:    32100 Batch Loss:     1.958156 Tokens per Sec:    22375, Lr: 0.000300\n",
      "2020-02-12 17:00:25,369 Epoch  14 Step:    32200 Batch Loss:     1.898870 Tokens per Sec:    21695, Lr: 0.000300\n",
      "2020-02-12 17:00:35,531 Epoch  14 Step:    32300 Batch Loss:     1.929601 Tokens per Sec:    21986, Lr: 0.000300\n",
      "2020-02-12 17:00:45,606 Epoch  14 Step:    32400 Batch Loss:     1.698034 Tokens per Sec:    21782, Lr: 0.000300\n",
      "2020-02-12 17:00:55,742 Epoch  14 Step:    32500 Batch Loss:     1.747295 Tokens per Sec:    21194, Lr: 0.000300\n",
      "2020-02-12 17:01:05,768 Epoch  14 Step:    32600 Batch Loss:     1.761056 Tokens per Sec:    21731, Lr: 0.000300\n",
      "2020-02-12 17:01:15,807 Epoch  14 Step:    32700 Batch Loss:     1.818480 Tokens per Sec:    22208, Lr: 0.000300\n",
      "2020-02-12 17:01:25,893 Epoch  14 Step:    32800 Batch Loss:     1.177521 Tokens per Sec:    21313, Lr: 0.000300\n",
      "2020-02-12 17:01:35,748 Epoch  14 Step:    32900 Batch Loss:     2.045574 Tokens per Sec:    22199, Lr: 0.000300\n",
      "2020-02-12 17:01:45,657 Epoch  14 Step:    33000 Batch Loss:     1.637881 Tokens per Sec:    22397, Lr: 0.000300\n",
      "2020-02-12 17:02:03,251 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:02:03,252 Saving new checkpoint.\n",
      "2020-02-12 17:02:03,441 Example #0\n",
      "2020-02-12 17:02:03,441 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:02:03,441 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:02:03,441 \tHypothesis: Er i lu Yehova a ver ivese i tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun ken dooshima u a lu ken dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:02:03,442 Example #1\n",
      "2020-02-12 17:02:03,442 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:02:03,442 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:02:03,442 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 17:02:03,442 Example #2\n",
      "2020-02-12 17:02:03,442 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:02:03,442 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:02:03,442 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken tar u Masedonia man Timoteu .\n",
      "2020-02-12 17:02:03,442 Example #3\n",
      "2020-02-12 17:02:03,442 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:02:03,443 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:02:03,443 \tHypothesis: Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:02:03,443 Validation result (greedy) at epoch  14, step    33000: bleu:  21.97, loss: 49051.1992, ppl:   5.2431, duration: 17.7851s\n",
      "2020-02-12 17:02:13,338 Epoch  14 Step:    33100 Batch Loss:     1.847190 Tokens per Sec:    22116, Lr: 0.000300\n",
      "2020-02-12 17:02:23,224 Epoch  14 Step:    33200 Batch Loss:     2.197025 Tokens per Sec:    22489, Lr: 0.000300\n",
      "2020-02-12 17:02:33,111 Epoch  14 Step:    33300 Batch Loss:     2.133088 Tokens per Sec:    22731, Lr: 0.000300\n",
      "2020-02-12 17:02:42,966 Epoch  14 Step:    33400 Batch Loss:     1.682475 Tokens per Sec:    22319, Lr: 0.000300\n",
      "2020-02-12 17:02:52,914 Epoch  14 Step:    33500 Batch Loss:     2.010394 Tokens per Sec:    22212, Lr: 0.000300\n",
      "2020-02-12 17:03:02,892 Epoch  14 Step:    33600 Batch Loss:     1.876231 Tokens per Sec:    22539, Lr: 0.000300\n",
      "2020-02-12 17:03:12,801 Epoch  14 Step:    33700 Batch Loss:     1.957483 Tokens per Sec:    22220, Lr: 0.000300\n",
      "2020-02-12 17:03:22,980 Epoch  14 Step:    33800 Batch Loss:     1.833876 Tokens per Sec:    22043, Lr: 0.000300\n",
      "2020-02-12 17:03:32,956 Epoch  14 Step:    33900 Batch Loss:     1.734484 Tokens per Sec:    22048, Lr: 0.000300\n",
      "2020-02-12 17:03:42,900 Epoch  14 Step:    34000 Batch Loss:     1.799194 Tokens per Sec:    22208, Lr: 0.000300\n",
      "2020-02-12 17:04:02,319 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:04:02,319 Saving new checkpoint.\n",
      "2020-02-12 17:04:02,510 Example #0\n",
      "2020-02-12 17:04:02,510 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:04:02,510 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:04:02,510 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu hen atô ve la , sha er vea seer doon ayol a ve yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:04:02,510 Example #1\n",
      "2020-02-12 17:04:02,510 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:04:02,510 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:04:02,511 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka hen shighe la je la ! — Rom .\n",
      "2020-02-12 17:04:02,511 Example #2\n",
      "2020-02-12 17:04:02,511 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:04:02,511 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:04:02,511 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:04:02,511 Example #3\n",
      "2020-02-12 17:04:02,511 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:04:02,511 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:04:02,511 \tHypothesis: Alaghga or nana lu a ian i eren kwagh sha inja ga .\n",
      "2020-02-12 17:04:02,511 Validation result (greedy) at epoch  14, step    34000: bleu:  22.95, loss: 48749.5977, ppl:   5.1899, duration: 19.6114s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:04:12,390 Epoch  14 Step:    34100 Batch Loss:     1.889835 Tokens per Sec:    22605, Lr: 0.000300\n",
      "2020-02-12 17:04:22,256 Epoch  14 Step:    34200 Batch Loss:     1.840223 Tokens per Sec:    22792, Lr: 0.000300\n",
      "2020-02-12 17:04:32,098 Epoch  14 Step:    34300 Batch Loss:     1.720049 Tokens per Sec:    22295, Lr: 0.000300\n",
      "2020-02-12 17:04:41,945 Epoch  14 Step:    34400 Batch Loss:     1.546757 Tokens per Sec:    22565, Lr: 0.000300\n",
      "2020-02-12 17:04:45,193 Epoch  14: total training loss 4325.08\n",
      "2020-02-12 17:04:45,193 EPOCH 15\n",
      "2020-02-12 17:04:52,155 Epoch  15 Step:    34500 Batch Loss:     1.768502 Tokens per Sec:    21324, Lr: 0.000300\n",
      "2020-02-12 17:05:02,101 Epoch  15 Step:    34600 Batch Loss:     1.467681 Tokens per Sec:    22141, Lr: 0.000300\n",
      "2020-02-12 17:05:12,021 Epoch  15 Step:    34700 Batch Loss:     1.651231 Tokens per Sec:    22103, Lr: 0.000300\n",
      "2020-02-12 17:05:21,987 Epoch  15 Step:    34800 Batch Loss:     1.513118 Tokens per Sec:    22192, Lr: 0.000300\n",
      "2020-02-12 17:05:31,949 Epoch  15 Step:    34900 Batch Loss:     1.697535 Tokens per Sec:    22015, Lr: 0.000300\n",
      "2020-02-12 17:05:42,042 Epoch  15 Step:    35000 Batch Loss:     1.861297 Tokens per Sec:    21823, Lr: 0.000300\n",
      "2020-02-12 17:06:01,337 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:06:01,337 Saving new checkpoint.\n",
      "2020-02-12 17:06:01,525 Example #0\n",
      "2020-02-12 17:06:01,525 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:06:01,525 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:06:01,525 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo tsung vea za hemen u lun a dooshima u ve lu a mi sha kwagh u dooshima la , shi vea za hemen u lun a dooshima u a lu a mi sha kwagh u dooshima la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:06:01,525 Example #1\n",
      "2020-02-12 17:06:01,526 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:06:01,526 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:06:01,526 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:06:01,526 Example #2\n",
      "2020-02-12 17:06:01,526 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:06:01,526 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:06:01,526 \tHypothesis: Ken anyom pue kar yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:06:01,526 Example #3\n",
      "2020-02-12 17:06:01,526 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:06:01,526 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:06:01,526 \tHypothesis: ( 1 Mbakorinte 15 : 1 ) Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:06:01,526 Validation result (greedy) at epoch  15, step    35000: bleu:  23.56, loss: 48539.9727, ppl:   5.1533, duration: 19.4843s\n",
      "2020-02-12 17:06:11,443 Epoch  15 Step:    35100 Batch Loss:     1.653404 Tokens per Sec:    22592, Lr: 0.000300\n",
      "2020-02-12 17:06:21,509 Epoch  15 Step:    35200 Batch Loss:     1.863294 Tokens per Sec:    22177, Lr: 0.000300\n",
      "2020-02-12 17:06:31,619 Epoch  15 Step:    35300 Batch Loss:     1.765835 Tokens per Sec:    21321, Lr: 0.000300\n",
      "2020-02-12 17:06:41,787 Epoch  15 Step:    35400 Batch Loss:     1.810177 Tokens per Sec:    21681, Lr: 0.000300\n",
      "2020-02-12 17:06:51,929 Epoch  15 Step:    35500 Batch Loss:     1.819806 Tokens per Sec:    21615, Lr: 0.000300\n",
      "2020-02-12 17:07:02,028 Epoch  15 Step:    35600 Batch Loss:     1.702587 Tokens per Sec:    22054, Lr: 0.000300\n",
      "2020-02-12 17:07:12,068 Epoch  15 Step:    35700 Batch Loss:     1.696817 Tokens per Sec:    22107, Lr: 0.000300\n",
      "2020-02-12 17:07:21,987 Epoch  15 Step:    35800 Batch Loss:     1.837978 Tokens per Sec:    22963, Lr: 0.000300\n",
      "2020-02-12 17:07:31,915 Epoch  15 Step:    35900 Batch Loss:     1.829086 Tokens per Sec:    22182, Lr: 0.000300\n",
      "2020-02-12 17:07:41,853 Epoch  15 Step:    36000 Batch Loss:     1.701076 Tokens per Sec:    22653, Lr: 0.000300\n",
      "2020-02-12 17:08:01,107 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:08:01,108 Saving new checkpoint.\n",
      "2020-02-12 17:08:01,299 Example #0\n",
      "2020-02-12 17:08:01,299 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:08:01,299 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:08:01,299 \tHypothesis: Er ivese i Yehova i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u mimi , shi vea za hemen u lun a dooshima u a lu hen atô ve la sha er dooshima una seer taver yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:08:01,299 Example #1\n",
      "2020-02-12 17:08:01,299 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:08:01,300 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:08:01,300 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka kwagh u a lu ikyua a vese je la ! — Rom .\n",
      "2020-02-12 17:08:01,300 Example #2\n",
      "2020-02-12 17:08:01,300 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:08:01,300 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:08:01,300 \tHypothesis: Anyom nga kar pue yô , Paulu ngu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:08:01,300 Example #3\n",
      "2020-02-12 17:08:01,300 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:08:01,301 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:08:01,301 \tHypothesis: Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:08:01,301 Validation result (greedy) at epoch  15, step    36000: bleu:  23.03, loss: 48129.7148, ppl:   5.0824, duration: 19.4477s\n",
      "2020-02-12 17:08:11,260 Epoch  15 Step:    36100 Batch Loss:     1.779255 Tokens per Sec:    22263, Lr: 0.000300\n",
      "2020-02-12 17:08:21,156 Epoch  15 Step:    36200 Batch Loss:     1.572476 Tokens per Sec:    22725, Lr: 0.000300\n",
      "2020-02-12 17:08:31,045 Epoch  15 Step:    36300 Batch Loss:     1.762110 Tokens per Sec:    22312, Lr: 0.000300\n",
      "2020-02-12 17:08:40,981 Epoch  15 Step:    36400 Batch Loss:     1.536011 Tokens per Sec:    21977, Lr: 0.000300\n",
      "2020-02-12 17:08:50,989 Epoch  15 Step:    36500 Batch Loss:     1.783622 Tokens per Sec:    22671, Lr: 0.000300\n",
      "2020-02-12 17:09:01,013 Epoch  15 Step:    36600 Batch Loss:     1.688564 Tokens per Sec:    21703, Lr: 0.000300\n",
      "2020-02-12 17:09:11,007 Epoch  15 Step:    36700 Batch Loss:     1.622414 Tokens per Sec:    22234, Lr: 0.000300\n",
      "2020-02-12 17:09:20,959 Epoch  15 Step:    36800 Batch Loss:     1.792515 Tokens per Sec:    21189, Lr: 0.000300\n",
      "2020-02-12 17:09:29,903 Epoch  15: total training loss 4265.62\n",
      "2020-02-12 17:09:29,904 EPOCH 16\n",
      "2020-02-12 17:09:31,058 Epoch  16 Step:    36900 Batch Loss:     1.520722 Tokens per Sec:    15740, Lr: 0.000300\n",
      "2020-02-12 17:09:40,884 Epoch  16 Step:    37000 Batch Loss:     1.663118 Tokens per Sec:    22476, Lr: 0.000300\n",
      "2020-02-12 17:10:00,117 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:10:00,117 Saving new checkpoint.\n",
      "2020-02-12 17:10:00,317 Example #0\n",
      "2020-02-12 17:10:00,318 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:10:00,318 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:10:00,318 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha kwagh u dooshima la , shi vea za hemen u lun a dooshima u a lu hen atô ve la , shi vea za hemen u lun a dooshima u a lu seer a seer la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:10:00,318 Example #1\n",
      "2020-02-12 17:10:00,318 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:10:00,318 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:10:00,318 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:10:00,318 Example #2\n",
      "2020-02-12 17:10:00,319 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:10:00,319 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:10:00,319 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:10:00,319 Example #3\n",
      "2020-02-12 17:10:00,319 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:10:00,319 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:10:00,319 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:10:00,319 Validation result (greedy) at epoch  16, step    37000: bleu:  23.29, loss: 47975.3398, ppl:   5.0560, duration: 19.4346s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:10:10,405 Epoch  16 Step:    37100 Batch Loss:     1.737724 Tokens per Sec:    21664, Lr: 0.000300\n",
      "2020-02-12 17:21:09,354 Epoch  18 Step:    42700 Batch Loss:     1.341680 Tokens per Sec:    22421, Lr: 0.000300\n",
      "2020-02-12 17:21:19,331 Epoch  18 Step:    42800 Batch Loss:     1.568339 Tokens per Sec:    22233, Lr: 0.000300\n",
      "2020-02-12 17:21:29,473 Epoch  18 Step:    42900 Batch Loss:     1.789617 Tokens per Sec:    21805, Lr: 0.000300\n",
      "2020-02-12 17:21:39,637 Epoch  18 Step:    43000 Batch Loss:     1.741903 Tokens per Sec:    21691, Lr: 0.000300\n",
      "2020-02-12 17:21:59,782 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:21:59,783 Saving new checkpoint.\n",
      "2020-02-12 17:21:59,974 Example #0\n",
      "2020-02-12 17:21:59,974 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:21:59,974 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:21:59,974 \tHypothesis: Er ivese i Yehova i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo kwagh kpoghuloo ve vea za hemen u lun a dooshima u mimi , shi vea za hemen u lun a dooshima u a lu hen atô ve la ye . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:21:59,974 Example #1\n",
      "2020-02-12 17:21:59,974 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:21:59,974 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:21:59,974 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:21:59,975 Example #2\n",
      "2020-02-12 17:21:59,975 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:21:59,975 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:21:59,975 \tHypothesis: Anyom nga kar pue yô , Paulu ngu ken Masedonia man Timoteu , Timoteu .\n",
      "2020-02-12 17:21:59,975 Example #3\n",
      "2020-02-12 17:21:59,975 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:21:59,975 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:21:59,975 \tHypothesis: Alaghga a lu or u nan lu a inja u eren kwagh sha inja ga yô .\n",
      "2020-02-12 17:21:59,975 Validation result (greedy) at epoch  18, step    43000: bleu:  24.27, loss: 46881.6602, ppl:   4.8726, duration: 20.3381s\n",
      "2020-02-12 17:22:09,952 Epoch  18 Step:    43100 Batch Loss:     1.719406 Tokens per Sec:    22462, Lr: 0.000300\n",
      "2020-02-12 17:22:19,828 Epoch  18 Step:    43200 Batch Loss:     1.828850 Tokens per Sec:    22917, Lr: 0.000300\n",
      "2020-02-12 17:22:29,803 Epoch  18 Step:    43300 Batch Loss:     1.869483 Tokens per Sec:    22392, Lr: 0.000300\n",
      "2020-02-12 17:22:39,889 Epoch  18 Step:    43400 Batch Loss:     1.468729 Tokens per Sec:    21585, Lr: 0.000300\n",
      "2020-02-12 17:22:49,860 Epoch  18 Step:    43500 Batch Loss:     1.788264 Tokens per Sec:    22357, Lr: 0.000300\n",
      "2020-02-12 17:22:59,685 Epoch  18 Step:    43600 Batch Loss:     1.795012 Tokens per Sec:    22602, Lr: 0.000300\n",
      "2020-02-12 17:23:09,544 Epoch  18 Step:    43700 Batch Loss:     1.944217 Tokens per Sec:    22824, Lr: 0.000300\n",
      "2020-02-12 17:23:19,380 Epoch  18 Step:    43800 Batch Loss:     1.707191 Tokens per Sec:    22251, Lr: 0.000300\n",
      "2020-02-12 17:23:29,289 Epoch  18 Step:    43900 Batch Loss:     1.899059 Tokens per Sec:    22764, Lr: 0.000300\n",
      "2020-02-12 17:23:39,179 Epoch  18 Step:    44000 Batch Loss:     1.937513 Tokens per Sec:    22186, Lr: 0.000300\n",
      "2020-02-12 17:23:58,677 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:23:58,677 Saving new checkpoint.\n",
      "2020-02-12 17:23:58,865 Example #0\n",
      "2020-02-12 17:23:58,865 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:23:58,866 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:23:58,866 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo kwagh kpoghuloo sha er vea kura dooshima ve u a lu hen atô ve la shi vea za hemen u lun a dooshima u a lu seer a seer la yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:23:58,866 Example #1\n",
      "2020-02-12 17:23:58,866 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:23:58,866 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:23:58,866 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:23:58,866 Example #2\n",
      "2020-02-12 17:23:58,866 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:23:58,866 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:23:58,866 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:23:58,866 Example #3\n",
      "2020-02-12 17:23:58,867 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:23:58,867 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:23:58,867 \tHypothesis: Alaghga a lu u or nana er kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:23:58,867 Validation result (greedy) at epoch  18, step    44000: bleu:  23.78, loss: 46783.2617, ppl:   4.8564, duration: 19.6870s\n",
      "2020-02-12 17:24:08,857 Epoch  18 Step:    44100 Batch Loss:     1.850072 Tokens per Sec:    22430, Lr: 0.000300\n",
      "2020-02-12 17:24:18,787 Epoch  18 Step:    44200 Batch Loss:     1.684474 Tokens per Sec:    21987, Lr: 0.000300\n",
      "2020-02-12 17:24:24,488 Epoch  18: total training loss 4123.80\n",
      "2020-02-12 17:24:24,489 EPOCH 19\n",
      "2020-02-12 17:24:29,005 Epoch  19 Step:    44300 Batch Loss:     2.051862 Tokens per Sec:    21713, Lr: 0.000300\n",
      "2020-02-12 17:24:39,064 Epoch  19 Step:    44400 Batch Loss:     1.527221 Tokens per Sec:    22102, Lr: 0.000300\n",
      "2020-02-12 17:24:49,197 Epoch  19 Step:    44500 Batch Loss:     1.698547 Tokens per Sec:    21756, Lr: 0.000300\n",
      "2020-02-12 17:24:59,313 Epoch  19 Step:    44600 Batch Loss:     1.678486 Tokens per Sec:    22101, Lr: 0.000300\n",
      "2020-02-12 17:25:09,400 Epoch  19 Step:    44700 Batch Loss:     1.522792 Tokens per Sec:    22018, Lr: 0.000300\n",
      "2020-02-12 17:25:19,338 Epoch  19 Step:    44800 Batch Loss:     1.547367 Tokens per Sec:    22460, Lr: 0.000300\n",
      "2020-02-12 17:25:29,252 Epoch  19 Step:    44900 Batch Loss:     2.023463 Tokens per Sec:    22413, Lr: 0.000300\n",
      "2020-02-12 17:25:39,210 Epoch  19 Step:    45000 Batch Loss:     1.765261 Tokens per Sec:    22156, Lr: 0.000300\n",
      "2020-02-12 17:25:57,565 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:25:57,565 Saving new checkpoint.\n",
      "2020-02-12 17:25:57,764 Example #0\n",
      "2020-02-12 17:25:57,764 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:25:57,764 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:25:57,764 \tHypothesis: Er Yehova a ver ivese i tsôron yô , gba hange hange u noov man kasev vea nôngo kwagh kpoghuloo ve vea za hemen u lun a dooshima u ve lu a mi sha kwagh u dooshima la , shi vea za hemen u lun a dooshima u a lu hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:25:57,764 Example #1\n",
      "2020-02-12 17:25:57,764 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:25:57,764 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:25:57,765 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe i doon tsung ne , man ka i lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:25:57,765 Example #2\n",
      "2020-02-12 17:25:57,765 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:25:57,765 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:25:57,765 \tHypothesis: Anyom nga kar pue yô , Paulu ngu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:25:57,765 Example #3\n",
      "2020-02-12 17:25:57,765 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:25:57,765 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:25:57,765 \tHypothesis: Alaghga a lu inja er or ngu eren kwagh sha inja ga nahan .\n",
      "2020-02-12 17:25:57,765 Validation result (greedy) at epoch  19, step    45000: bleu:  24.12, loss: 46635.3867, ppl:   4.8322, duration: 18.5550s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:26:07,696 Epoch  19 Step:    45100 Batch Loss:     1.836917 Tokens per Sec:    22059, Lr: 0.000300\n",
      "2020-02-12 17:26:17,646 Epoch  19 Step:    45200 Batch Loss:     1.467316 Tokens per Sec:    22531, Lr: 0.000300\n",
      "2020-02-12 17:26:27,623 Epoch  19 Step:    45300 Batch Loss:     1.681551 Tokens per Sec:    22316, Lr: 0.000300\n",
      "2020-02-12 17:26:37,794 Epoch  19 Step:    45400 Batch Loss:     1.638225 Tokens per Sec:    22062, Lr: 0.000300\n",
      "2020-02-12 17:26:47,800 Epoch  19 Step:    45500 Batch Loss:     1.648621 Tokens per Sec:    21838, Lr: 0.000300\n",
      "2020-02-12 17:26:57,727 Epoch  19 Step:    45600 Batch Loss:     1.807967 Tokens per Sec:    21813, Lr: 0.000300\n",
      "2020-02-12 17:27:07,666 Epoch  19 Step:    45700 Batch Loss:     1.725803 Tokens per Sec:    22564, Lr: 0.000300\n",
      "2020-02-12 17:27:17,616 Epoch  19 Step:    45800 Batch Loss:     1.618219 Tokens per Sec:    22026, Lr: 0.000300\n",
      "2020-02-12 17:27:27,727 Epoch  19 Step:    45900 Batch Loss:     1.445169 Tokens per Sec:    21406, Lr: 0.000300\n",
      "2020-02-12 17:27:37,830 Epoch  19 Step:    46000 Batch Loss:     1.583930 Tokens per Sec:    22123, Lr: 0.000300\n",
      "2020-02-12 17:27:56,600 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:27:56,600 Saving new checkpoint.\n",
      "2020-02-12 17:27:56,790 Example #0\n",
      "2020-02-12 17:27:56,791 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:27:56,791 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:27:56,791 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo vea za hemen u lun a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u a lu a mi sha dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:27:56,791 Example #1\n",
      "2020-02-12 17:27:56,791 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:27:56,791 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:27:56,791 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:27:56,791 Example #2\n",
      "2020-02-12 17:27:56,792 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:27:56,792 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:27:56,792 \tHypothesis: Anyom nga kar pue yô , Paulu ngu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:27:56,792 Example #3\n",
      "2020-02-12 17:27:56,792 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:27:56,792 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:27:56,792 \tHypothesis: Alaghga or nana lu a ian i eren kwagh u vough ga .\n",
      "2020-02-12 17:27:56,792 Validation result (greedy) at epoch  19, step    46000: bleu:  24.22, loss: 46367.6172, ppl:   4.7887, duration: 18.9613s\n",
      "2020-02-12 17:28:06,980 Epoch  19 Step:    46100 Batch Loss:     1.860646 Tokens per Sec:    21371, Lr: 0.000300\n",
      "2020-02-12 17:38:57,600 Epoch  21: total training loss 4001.43\n",
      "2020-02-12 17:38:57,600 EPOCH 22\n",
      "2020-02-12 17:39:04,843 Epoch  22 Step:    51700 Batch Loss:     1.772589 Tokens per Sec:    21140, Lr: 0.000300\n",
      "2020-02-12 17:39:14,757 Epoch  22 Step:    51800 Batch Loss:     1.617656 Tokens per Sec:    22721, Lr: 0.000300\n",
      "2020-02-12 17:39:24,745 Epoch  22 Step:    51900 Batch Loss:     1.723209 Tokens per Sec:    22303, Lr: 0.000300\n",
      "2020-02-12 17:39:34,599 Epoch  22 Step:    52000 Batch Loss:     1.459682 Tokens per Sec:    22129, Lr: 0.000300\n",
      "2020-02-12 17:39:52,655 Example #0\n",
      "2020-02-12 17:39:52,655 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:39:52,656 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:39:52,656 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo tsung vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:39:52,656 Example #1\n",
      "2020-02-12 17:39:52,656 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:39:52,656 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:39:52,656 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka hen shighe la je la ! — Rom .\n",
      "2020-02-12 17:39:52,656 Example #2\n",
      "2020-02-12 17:39:52,656 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:39:52,656 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:39:52,656 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:39:52,656 Example #3\n",
      "2020-02-12 17:39:52,657 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:39:52,657 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:39:52,657 \tHypothesis: Alaghga a lu kwagh u or nana er sha gbenda u vough ga yô .\n",
      "2020-02-12 17:39:52,657 Validation result (greedy) at epoch  22, step    52000: bleu:  24.00, loss: 45752.3438, ppl:   4.6902, duration: 18.0572s\n",
      "2020-02-12 17:40:02,579 Epoch  22 Step:    52100 Batch Loss:     1.620099 Tokens per Sec:    22036, Lr: 0.000300\n",
      "2020-02-12 17:40:12,508 Epoch  22 Step:    52200 Batch Loss:     1.498319 Tokens per Sec:    23034, Lr: 0.000300\n",
      "2020-02-12 17:40:22,392 Epoch  22 Step:    52300 Batch Loss:     1.437057 Tokens per Sec:    22007, Lr: 0.000300\n",
      "2020-02-12 17:40:32,403 Epoch  22 Step:    52400 Batch Loss:     1.926183 Tokens per Sec:    22309, Lr: 0.000300\n",
      "2020-02-12 17:40:42,282 Epoch  22 Step:    52500 Batch Loss:     1.597002 Tokens per Sec:    22219, Lr: 0.000300\n",
      "2020-02-12 17:40:52,247 Epoch  22 Step:    52600 Batch Loss:     1.793759 Tokens per Sec:    22097, Lr: 0.000300\n",
      "2020-02-12 17:41:02,173 Epoch  22 Step:    52700 Batch Loss:     1.272365 Tokens per Sec:    22586, Lr: 0.000300\n",
      "2020-02-12 17:41:12,267 Epoch  22 Step:    52800 Batch Loss:     1.542842 Tokens per Sec:    22351, Lr: 0.000300\n",
      "2020-02-12 17:41:22,402 Epoch  22 Step:    52900 Batch Loss:     1.864567 Tokens per Sec:    22087, Lr: 0.000300\n",
      "2020-02-12 17:41:32,491 Epoch  22 Step:    53000 Batch Loss:     1.595312 Tokens per Sec:    21344, Lr: 0.000300\n",
      "2020-02-12 17:41:50,893 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:41:50,893 Saving new checkpoint.\n",
      "2020-02-12 17:41:51,098 Example #0\n",
      "2020-02-12 17:41:51,098 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:41:51,099 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:41:51,099 \tHypothesis: Er Yehova a ver ivese i tsôron yô , gba hange hange u noov man kasev vea eren kwagh sha dooshima shi vea za hemen u lun a dooshima u mimi sha dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:41:51,099 Example #1\n",
      "2020-02-12 17:41:51,099 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:41:51,099 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:41:51,099 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:41:51,099 Example #2\n",
      "2020-02-12 17:41:51,099 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:41:51,099 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:41:51,099 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:41:51,100 Example #3\n",
      "2020-02-12 17:41:51,100 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:41:51,100 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:41:51,100 \tHypothesis: Alaghga or nana lu eren kwagh sha inja ga .\n",
      "2020-02-12 17:41:51,100 Validation result (greedy) at epoch  22, step    53000: bleu:  24.75, loss: 45526.4297, ppl:   4.6546, duration: 18.6082s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:42:01,191 Epoch  22 Step:    53100 Batch Loss:     1.735585 Tokens per Sec:    22267, Lr: 0.000300\n",
      "2020-02-12 17:42:11,271 Epoch  22 Step:    53200 Batch Loss:     1.616482 Tokens per Sec:    21461, Lr: 0.000300\n",
      "2020-02-12 17:42:21,408 Epoch  22 Step:    53300 Batch Loss:     1.763771 Tokens per Sec:    21460, Lr: 0.000300\n",
      "2020-02-12 17:42:31,417 Epoch  22 Step:    53400 Batch Loss:     1.682316 Tokens per Sec:    21922, Lr: 0.000300\n",
      "2020-02-12 17:42:41,387 Epoch  22 Step:    53500 Batch Loss:     1.623947 Tokens per Sec:    22414, Lr: 0.000300\n",
      "2020-02-12 17:42:51,340 Epoch  22 Step:    53600 Batch Loss:     1.663797 Tokens per Sec:    22639, Lr: 0.000300\n",
      "2020-02-12 17:43:01,235 Epoch  22 Step:    53700 Batch Loss:     1.489120 Tokens per Sec:    22257, Lr: 0.000300\n",
      "2020-02-12 17:43:11,193 Epoch  22 Step:    53800 Batch Loss:     1.619790 Tokens per Sec:    22105, Lr: 0.000300\n",
      "2020-02-12 17:43:21,083 Epoch  22 Step:    53900 Batch Loss:     1.598196 Tokens per Sec:    22197, Lr: 0.000300\n",
      "2020-02-12 17:43:30,936 Epoch  22 Step:    54000 Batch Loss:     1.607882 Tokens per Sec:    22312, Lr: 0.000300\n",
      "2020-02-12 17:43:49,878 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:43:49,878 Saving new checkpoint.\n",
      "2020-02-12 17:43:50,065 Example #0\n",
      "2020-02-12 17:43:50,066 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:43:50,066 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:43:50,066 \tHypothesis: Er i ver ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea eren kwagh sha dooshima ve , shi vea za hemen u lun a dooshima u a lu a mi sha kwagh u dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:43:50,066 Example #1\n",
      "2020-02-12 17:43:50,066 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:43:50,066 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:43:50,066 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 17:43:50,066 Example #2\n",
      "2020-02-12 17:43:50,067 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:43:50,067 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:43:50,067 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken tar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:43:50,067 Example #3\n",
      "2020-02-12 17:43:50,067 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:43:50,067 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:43:50,067 \tHypothesis: Alaghga a lu or u nan lu eren kwagh sha inja ga la .\n",
      "2020-02-12 17:43:50,067 Validation result (greedy) at epoch  22, step    54000: bleu:  24.68, loss: 45278.1523, ppl:   4.6157, duration: 19.1309s\n",
      "2020-02-12 17:43:58,658 Epoch  22: total training loss 3981.50\n",
      "2020-02-12 17:43:58,658 EPOCH 23\n",
      "2020-02-12 17:44:00,213 Epoch  23 Step:    54100 Batch Loss:     1.778395 Tokens per Sec:    18414, Lr: 0.000300\n",
      "2020-02-12 17:44:10,129 Epoch  23 Step:    54200 Batch Loss:     1.823727 Tokens per Sec:    22199, Lr: 0.000300\n",
      "2020-02-12 17:44:20,124 Epoch  23 Step:    54300 Batch Loss:     1.714084 Tokens per Sec:    22124, Lr: 0.000300\n",
      "2020-02-12 17:44:30,180 Epoch  23 Step:    54400 Batch Loss:     1.776259 Tokens per Sec:    21936, Lr: 0.000300\n",
      "2020-02-12 17:44:40,212 Epoch  23 Step:    54500 Batch Loss:     1.771630 Tokens per Sec:    22107, Lr: 0.000300\n",
      "2020-02-12 17:44:50,362 Epoch  23 Step:    54600 Batch Loss:     1.512814 Tokens per Sec:    21790, Lr: 0.000300\n",
      "2020-02-12 17:45:00,211 Epoch  23 Step:    54700 Batch Loss:     1.489588 Tokens per Sec:    22399, Lr: 0.000300\n",
      "2020-02-12 17:45:10,116 Epoch  23 Step:    54800 Batch Loss:     1.749939 Tokens per Sec:    22225, Lr: 0.000300\n",
      "2020-02-12 17:45:20,170 Epoch  23 Step:    54900 Batch Loss:     1.765753 Tokens per Sec:    22497, Lr: 0.000300\n",
      "2020-02-12 17:45:30,023 Epoch  23 Step:    55000 Batch Loss:     1.529017 Tokens per Sec:    22112, Lr: 0.000300\n",
      "2020-02-12 17:45:49,033 Example #0\n",
      "2020-02-12 17:45:49,033 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:45:49,033 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:45:49,033 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo tsung vea kura dooshima ve u a lu hen atô ve la , shi vea za hemen u lun a dooshima u a lu a mi sha dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:45:49,033 Example #1\n",
      "2020-02-12 17:45:49,034 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:45:49,034 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:45:49,034 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka hen shighe la je la ! — Rom .\n",
      "2020-02-12 17:45:49,034 Example #2\n",
      "2020-02-12 17:45:49,034 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:45:49,034 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:45:49,034 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:45:49,034 Example #3\n",
      "2020-02-12 17:45:49,035 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:45:49,035 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:45:49,035 \tHypothesis: Alaghga a lu inja er or ngu eren kwagh sha inja ga nahan .\n",
      "2020-02-12 17:45:49,035 Validation result (greedy) at epoch  23, step    55000: bleu:  24.65, loss: 45355.6992, ppl:   4.6278, duration: 19.0111s\n",
      "2020-02-12 17:45:59,146 Epoch  23 Step:    55100 Batch Loss:     1.749363 Tokens per Sec:    21467, Lr: 0.000300\n",
      "2020-02-12 17:46:09,273 Epoch  23 Step:    55200 Batch Loss:     1.484830 Tokens per Sec:    21644, Lr: 0.000300\n",
      "2020-02-12 17:46:19,353 Epoch  23 Step:    55300 Batch Loss:     1.692046 Tokens per Sec:    21685, Lr: 0.000300\n",
      "2020-02-12 17:46:29,446 Epoch  23 Step:    55400 Batch Loss:     1.548436 Tokens per Sec:    21393, Lr: 0.000300\n",
      "2020-02-12 17:46:39,560 Epoch  23 Step:    55500 Batch Loss:     1.883074 Tokens per Sec:    21931, Lr: 0.000300\n",
      "2020-02-12 17:46:49,711 Epoch  23 Step:    55600 Batch Loss:     1.489641 Tokens per Sec:    22161, Lr: 0.000300\n",
      "2020-02-12 17:46:59,628 Epoch  23 Step:    55700 Batch Loss:     1.780797 Tokens per Sec:    21625, Lr: 0.000300\n",
      "2020-02-12 17:47:09,668 Epoch  23 Step:    55800 Batch Loss:     1.770255 Tokens per Sec:    22375, Lr: 0.000300\n",
      "2020-02-12 17:47:19,667 Epoch  23 Step:    55900 Batch Loss:     1.811267 Tokens per Sec:    23154, Lr: 0.000300\n",
      "2020-02-12 17:47:29,807 Epoch  23 Step:    56000 Batch Loss:     1.493290 Tokens per Sec:    21064, Lr: 0.000300\n",
      "2020-02-12 17:47:49,597 Example #0\n",
      "2020-02-12 17:47:49,597 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:47:49,597 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:47:49,598 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , doo u noov man kasev vea eren kwagh sha dooshima , shi vea za hemen u lun a dooshima u a lu hen atô ve la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:47:49,598 Example #1\n",
      "2020-02-12 17:47:49,598 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:47:49,598 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:47:49,598 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka shighe u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:47:49,598 Example #2\n",
      "2020-02-12 17:47:49,598 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:47:49,598 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:47:49,599 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu .\n",
      "2020-02-12 17:47:49,599 Example #3\n",
      "2020-02-12 17:47:49,599 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:47:49,599 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:47:49,599 \tHypothesis: Alaghga a lu u or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:47:49,599 Validation result (greedy) at epoch  23, step    56000: bleu:  25.10, loss: 45490.6133, ppl:   4.6489, duration: 19.7918s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:47:59,574 Epoch  23 Step:    56100 Batch Loss:     1.637856 Tokens per Sec:    22497, Lr: 0.000300\n",
      "2020-02-12 17:48:09,625 Epoch  23 Step:    56200 Batch Loss:     1.865298 Tokens per Sec:    22097, Lr: 0.000300\n",
      "2020-02-12 17:48:19,832 Epoch  23 Step:    56300 Batch Loss:     1.460898 Tokens per Sec:    21439, Lr: 0.000300\n",
      "2020-02-12 17:48:30,046 Epoch  23 Step:    56400 Batch Loss:     1.146451 Tokens per Sec:    21982, Lr: 0.000300\n",
      "2020-02-12 17:48:40,108 Epoch  23 Step:    56500 Batch Loss:     1.336150 Tokens per Sec:    21597, Lr: 0.000300\n",
      "2020-02-12 17:48:44,957 Epoch  23: total training loss 3955.45\n",
      "2020-02-12 17:48:44,957 EPOCH 24\n",
      "2020-02-12 17:48:50,541 Epoch  24 Step:    56600 Batch Loss:     1.687715 Tokens per Sec:    21734, Lr: 0.000300\n",
      "2020-02-12 17:49:00,481 Epoch  24 Step:    56700 Batch Loss:     1.656443 Tokens per Sec:    22469, Lr: 0.000300\n",
      "2020-02-12 17:49:10,669 Epoch  24 Step:    56800 Batch Loss:     1.551410 Tokens per Sec:    22112, Lr: 0.000300\n",
      "2020-02-12 17:49:20,801 Epoch  24 Step:    56900 Batch Loss:     1.403160 Tokens per Sec:    21329, Lr: 0.000300\n",
      "2020-02-12 17:49:30,755 Epoch  24 Step:    57000 Batch Loss:     1.746896 Tokens per Sec:    21962, Lr: 0.000300\n",
      "2020-02-12 17:49:49,871 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:49:49,871 Saving new checkpoint.\n",
      "2020-02-12 17:49:50,073 Example #0\n",
      "2020-02-12 17:49:50,073 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:49:50,073 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:49:50,073 \tHypothesis: Er Yehova a ver ivese i tsôron yô , doo u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu hen atô ve la , shi vea za hemen u lun a dooshima u a lu hen atô ve la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:49:50,073 Example #1\n",
      "2020-02-12 17:49:50,073 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:49:50,073 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:49:50,074 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 17:49:50,074 Example #2\n",
      "2020-02-12 17:49:50,074 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:49:50,074 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:49:50,074 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:49:50,074 Example #3\n",
      "2020-02-12 17:49:50,074 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:49:50,074 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:49:50,074 \tHypothesis: Alaghga a lu inja er or ngu eren kwagh sha inja ga nahan .\n",
      "2020-02-12 17:49:50,074 Validation result (greedy) at epoch  24, step    57000: bleu:  25.22, loss: 45200.4805, ppl:   4.6036, duration: 19.3191s\n",
      "2020-02-12 17:50:00,028 Epoch  24 Step:    57100 Batch Loss:     1.525229 Tokens per Sec:    21687, Lr: 0.000300\n",
      "2020-02-12 17:50:09,925 Epoch  24 Step:    57200 Batch Loss:     1.442598 Tokens per Sec:    22190, Lr: 0.000300\n",
      "2020-02-12 17:50:19,865 Epoch  24 Step:    57300 Batch Loss:     1.288021 Tokens per Sec:    22060, Lr: 0.000300\n",
      "2020-02-12 17:50:29,754 Epoch  24 Step:    57400 Batch Loss:     1.621453 Tokens per Sec:    22618, Lr: 0.000300\n",
      "2020-02-12 17:50:39,613 Epoch  24 Step:    57500 Batch Loss:     1.562569 Tokens per Sec:    22025, Lr: 0.000300\n",
      "2020-02-12 17:50:49,594 Epoch  24 Step:    57600 Batch Loss:     1.553372 Tokens per Sec:    22142, Lr: 0.000300\n",
      "2020-02-12 17:50:59,615 Epoch  24 Step:    57700 Batch Loss:     1.674770 Tokens per Sec:    22229, Lr: 0.000300\n",
      "2020-02-12 17:51:09,608 Epoch  24 Step:    57800 Batch Loss:     1.513857 Tokens per Sec:    22287, Lr: 0.000300\n",
      "2020-02-12 17:51:19,660 Epoch  24 Step:    57900 Batch Loss:     1.618932 Tokens per Sec:    21626, Lr: 0.000300\n",
      "2020-02-12 17:51:29,770 Epoch  24 Step:    58000 Batch Loss:     1.554007 Tokens per Sec:    21367, Lr: 0.000300\n",
      "2020-02-12 17:51:49,129 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:51:49,129 Saving new checkpoint.\n",
      "2020-02-12 17:51:49,320 Example #0\n",
      "2020-02-12 17:51:49,320 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:51:49,320 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:51:49,320 \tHypothesis: Er ivese i Yehova i ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha kwagh u dooshima la , shi vea za hemen u lun a dooshima u a lu a mi sha kwagh u dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:51:49,320 Example #1\n",
      "2020-02-12 17:51:49,320 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:51:49,321 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:51:49,321 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe i doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 17:51:49,321 Example #2\n",
      "2020-02-12 17:51:49,321 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:51:49,321 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:51:49,321 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken tar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:51:49,321 Example #3\n",
      "2020-02-12 17:51:49,321 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:51:49,321 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:51:49,321 \tHypothesis: Alaghga a lu sha gbenda u or nan lu a inja ga yô .\n",
      "2020-02-12 17:51:49,321 Validation result (greedy) at epoch  24, step    58000: bleu:  25.30, loss: 44995.5547, ppl:   4.5718, duration: 19.5509s\n",
      "2020-02-12 17:51:59,469 Epoch  24 Step:    58100 Batch Loss:     1.709372 Tokens per Sec:    21885, Lr: 0.000300\n",
      "2020-02-12 17:52:09,586 Epoch  24 Step:    58200 Batch Loss:     1.485827 Tokens per Sec:    21601, Lr: 0.000300\n",
      "2020-02-12 17:52:19,713 Epoch  24 Step:    58300 Batch Loss:     1.989177 Tokens per Sec:    21577, Lr: 0.000300\n",
      "2020-02-12 17:52:29,856 Epoch  24 Step:    58400 Batch Loss:     1.345109 Tokens per Sec:    21960, Lr: 0.000300\n",
      "2020-02-12 17:52:39,974 Epoch  24 Step:    58500 Batch Loss:     1.654352 Tokens per Sec:    22383, Lr: 0.000300\n",
      "2020-02-12 17:52:50,157 Epoch  24 Step:    58600 Batch Loss:     1.736848 Tokens per Sec:    21951, Lr: 0.000300\n",
      "2020-02-12 17:53:00,395 Epoch  24 Step:    58700 Batch Loss:     1.513085 Tokens per Sec:    20864, Lr: 0.000300\n",
      "2020-02-12 17:53:10,402 Epoch  24 Step:    58800 Batch Loss:     1.369668 Tokens per Sec:    22031, Lr: 0.000300\n",
      "2020-02-12 17:53:20,334 Epoch  24 Step:    58900 Batch Loss:     1.778354 Tokens per Sec:    22127, Lr: 0.000300\n",
      "2020-02-12 17:53:30,307 Epoch  24 Step:    59000 Batch Loss:     1.571671 Tokens per Sec:    22080, Lr: 0.000300\n",
      "2020-02-12 17:53:48,726 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:53:48,726 Saving new checkpoint.\n",
      "2020-02-12 17:53:48,916 Example #0\n",
      "2020-02-12 17:53:48,916 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:53:48,916 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:53:48,916 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo vea kura ayol a ve sha er dooshima ve una seer taver yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:53:48,916 Example #1\n",
      "2020-02-12 17:53:48,916 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:53:48,916 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:53:48,917 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka hen shighe la je la ! — Rom .\n",
      "2020-02-12 17:53:48,917 Example #2\n",
      "2020-02-12 17:53:48,917 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:53:48,917 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:53:48,917 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:53:48,917 Example #3\n",
      "2020-02-12 17:53:48,917 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:53:48,917 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:53:48,917 \tHypothesis: Alaghga a lu inja er or ngu eren kwagh sha inja ga nahan .\n",
      "2020-02-12 17:53:48,917 Validation result (greedy) at epoch  24, step    59000: bleu:  25.46, loss: 44955.9961, ppl:   4.5657, duration: 18.6104s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:53:50,207 Epoch  24: total training loss 3936.89\n",
      "2020-02-12 17:53:50,207 EPOCH 25\n",
      "2020-02-12 17:53:59,083 Epoch  25 Step:    59100 Batch Loss:     1.634622 Tokens per Sec:    22183, Lr: 0.000300\n",
      "2020-02-12 17:54:09,165 Epoch  25 Step:    59200 Batch Loss:     1.739247 Tokens per Sec:    21693, Lr: 0.000300\n",
      "2020-02-12 17:54:19,316 Epoch  25 Step:    59300 Batch Loss:     1.341596 Tokens per Sec:    21416, Lr: 0.000300\n",
      "2020-02-12 17:54:29,324 Epoch  25 Step:    59400 Batch Loss:     1.790190 Tokens per Sec:    22408, Lr: 0.000300\n",
      "2020-02-12 17:54:39,219 Epoch  25 Step:    59500 Batch Loss:     1.507016 Tokens per Sec:    22334, Lr: 0.000300\n",
      "2020-02-12 17:54:49,189 Epoch  25 Step:    59600 Batch Loss:     1.599367 Tokens per Sec:    22329, Lr: 0.000300\n",
      "2020-02-12 17:54:59,172 Epoch  25 Step:    59700 Batch Loss:     1.695844 Tokens per Sec:    22224, Lr: 0.000300\n",
      "2020-02-12 17:55:09,456 Epoch  25 Step:    59800 Batch Loss:     1.547237 Tokens per Sec:    21382, Lr: 0.000300\n",
      "2020-02-12 17:55:19,584 Epoch  25 Step:    59900 Batch Loss:     1.655974 Tokens per Sec:    21733, Lr: 0.000300\n",
      "2020-02-12 17:55:29,671 Epoch  25 Step:    60000 Batch Loss:     1.740964 Tokens per Sec:    22243, Lr: 0.000300\n",
      "2020-02-12 17:55:48,439 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:55:48,440 Saving new checkpoint.\n",
      "2020-02-12 17:55:48,631 Example #0\n",
      "2020-02-12 17:55:48,631 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:55:48,631 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:55:48,631 \tHypothesis: Er ivese i Yehova i lu kwagh u vesen yô , gba hange hange u noov man kasev vea eren kwagh sha inja sha inja sha u tesen dooshima ve shi vea za hemen u lun ken mzough u dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:55:48,631 Example #1\n",
      "2020-02-12 17:55:48,632 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:55:48,632 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:55:48,632 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 17:55:48,632 Example #2\n",
      "2020-02-12 17:55:48,632 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:55:48,632 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:55:48,632 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu .\n",
      "2020-02-12 17:55:48,632 Example #3\n",
      "2020-02-12 17:55:48,632 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:55:48,632 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:55:48,632 \tHypothesis: Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:55:48,633 Validation result (greedy) at epoch  25, step    60000: bleu:  25.21, loss: 44820.8516, ppl:   4.5449, duration: 18.9614s\n",
      "2020-02-12 17:55:58,628 Epoch  25 Step:    60100 Batch Loss:     1.556117 Tokens per Sec:    22396, Lr: 0.000300\n",
      "2020-02-12 17:56:08,549 Epoch  25 Step:    60200 Batch Loss:     1.435483 Tokens per Sec:    22393, Lr: 0.000300\n",
      "2020-02-12 17:56:18,455 Epoch  25 Step:    60300 Batch Loss:     1.532294 Tokens per Sec:    22111, Lr: 0.000300\n",
      "2020-02-12 17:56:28,341 Epoch  25 Step:    60400 Batch Loss:     1.790889 Tokens per Sec:    22252, Lr: 0.000300\n",
      "2020-02-12 17:56:38,227 Epoch  25 Step:    60500 Batch Loss:     1.527169 Tokens per Sec:    22305, Lr: 0.000300\n",
      "2020-02-12 17:56:48,170 Epoch  25 Step:    60600 Batch Loss:     1.493572 Tokens per Sec:    22785, Lr: 0.000300\n",
      "2020-02-12 17:56:58,131 Epoch  25 Step:    60700 Batch Loss:     1.356093 Tokens per Sec:    22259, Lr: 0.000300\n",
      "2020-02-12 17:57:08,096 Epoch  25 Step:    60800 Batch Loss:     1.618109 Tokens per Sec:    22001, Lr: 0.000300\n",
      "2020-02-12 17:57:18,036 Epoch  25 Step:    60900 Batch Loss:     1.624931 Tokens per Sec:    22428, Lr: 0.000300\n",
      "2020-02-12 17:57:28,191 Epoch  25 Step:    61000 Batch Loss:     1.683243 Tokens per Sec:    22334, Lr: 0.000300\n",
      "2020-02-12 17:57:47,907 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 17:57:47,907 Saving new checkpoint.\n",
      "2020-02-12 17:57:48,095 Example #0\n",
      "2020-02-12 17:57:48,096 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:57:48,096 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:57:48,096 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea kuran atindi a ve a dooshima la shi vea za hemen u lun a dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:57:48,096 Example #1\n",
      "2020-02-12 17:57:48,096 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:57:48,096 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:57:48,096 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 17:57:48,096 Example #2\n",
      "2020-02-12 17:57:48,097 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:57:48,097 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:57:48,097 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:57:48,097 Example #3\n",
      "2020-02-12 17:57:48,097 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:57:48,097 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:57:48,097 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:57:48,097 Validation result (greedy) at epoch  25, step    61000: bleu:  25.55, loss: 44636.4297, ppl:   4.5167, duration: 19.9060s\n",
      "2020-02-12 17:57:58,201 Epoch  25 Step:    61100 Batch Loss:     1.295469 Tokens per Sec:    21274, Lr: 0.000300\n",
      "2020-02-12 17:58:08,331 Epoch  25 Step:    61200 Batch Loss:     1.570902 Tokens per Sec:    21674, Lr: 0.000300\n",
      "2020-02-12 17:58:18,412 Epoch  25 Step:    61300 Batch Loss:     1.705322 Tokens per Sec:    21730, Lr: 0.000300\n",
      "2020-02-12 17:58:28,336 Epoch  25 Step:    61400 Batch Loss:     1.815144 Tokens per Sec:    22169, Lr: 0.000300\n",
      "2020-02-12 17:58:35,362 Epoch  25: total training loss 3896.86\n",
      "2020-02-12 17:58:35,362 EPOCH 26\n",
      "2020-02-12 17:58:38,517 Epoch  26 Step:    61500 Batch Loss:     1.476855 Tokens per Sec:    20573, Lr: 0.000300\n",
      "2020-02-12 17:58:48,521 Epoch  26 Step:    61600 Batch Loss:     1.359429 Tokens per Sec:    22378, Lr: 0.000300\n",
      "2020-02-12 17:58:58,475 Epoch  26 Step:    61700 Batch Loss:     1.735080 Tokens per Sec:    22423, Lr: 0.000300\n",
      "2020-02-12 17:59:08,391 Epoch  26 Step:    61800 Batch Loss:     1.506324 Tokens per Sec:    21819, Lr: 0.000300\n",
      "2020-02-12 17:59:18,404 Epoch  26 Step:    61900 Batch Loss:     1.664138 Tokens per Sec:    21684, Lr: 0.000300\n",
      "2020-02-12 17:59:28,346 Epoch  26 Step:    62000 Batch Loss:     1.762046 Tokens per Sec:    22158, Lr: 0.000300\n",
      "2020-02-12 17:59:47,553 Example #0\n",
      "2020-02-12 17:59:47,554 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 17:59:47,554 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 17:59:47,554 \tHypothesis: Er ivese i Yehova i ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 17:59:47,554 Example #1\n",
      "2020-02-12 17:59:47,554 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 17:59:47,554 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 17:59:47,554 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 17:59:47,554 Example #2\n",
      "2020-02-12 17:59:47,554 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 17:59:47,554 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 17:59:47,554 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 17:59:47,555 Example #3\n",
      "2020-02-12 17:59:47,555 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 17:59:47,555 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 17:59:47,555 \tHypothesis: Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 17:59:47,555 Validation result (greedy) at epoch  26, step    62000: bleu:  25.27, loss: 44655.5625, ppl:   4.5196, duration: 19.2080s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 17:59:57,420 Epoch  26 Step:    62100 Batch Loss:     1.538173 Tokens per Sec:    22215, Lr: 0.000300\n",
      "2020-02-12 18:00:07,278 Epoch  26 Step:    62200 Batch Loss:     1.745478 Tokens per Sec:    22920, Lr: 0.000300\n",
      "2020-02-12 18:00:17,139 Epoch  26 Step:    62300 Batch Loss:     1.611246 Tokens per Sec:    22182, Lr: 0.000300\n",
      "2020-02-12 18:00:27,120 Epoch  26 Step:    62400 Batch Loss:     1.708399 Tokens per Sec:    21687, Lr: 0.000300\n",
      "2020-02-12 18:00:36,992 Epoch  26 Step:    62500 Batch Loss:     1.464080 Tokens per Sec:    22382, Lr: 0.000300\n",
      "2020-02-12 18:00:46,971 Epoch  26 Step:    62600 Batch Loss:     1.664190 Tokens per Sec:    22094, Lr: 0.000300\n",
      "2020-02-12 18:00:56,913 Epoch  26 Step:    62700 Batch Loss:     1.557987 Tokens per Sec:    22267, Lr: 0.000300\n",
      "2020-02-12 18:01:06,841 Epoch  26 Step:    62800 Batch Loss:     1.668654 Tokens per Sec:    22139, Lr: 0.000300\n",
      "2020-02-12 18:01:16,817 Epoch  26 Step:    62900 Batch Loss:     1.345859 Tokens per Sec:    22119, Lr: 0.000300\n",
      "2020-02-12 18:01:26,785 Epoch  26 Step:    63000 Batch Loss:     1.626277 Tokens per Sec:    22379, Lr: 0.000300\n",
      "2020-02-12 18:01:45,212 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:01:45,212 Saving new checkpoint.\n",
      "2020-02-12 18:01:45,400 Example #0\n",
      "2020-02-12 18:01:45,400 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:01:45,400 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:01:45,400 \tHypothesis: Er Yehova a ver ivese i lun gbem sha won yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun ken dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:01:45,400 Example #1\n",
      "2020-02-12 18:01:45,400 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:01:45,400 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:01:45,401 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 18:01:45,401 Example #2\n",
      "2020-02-12 18:01:45,401 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:01:45,401 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:01:45,401 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:01:45,401 Example #3\n",
      "2020-02-12 18:01:45,401 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:01:45,401 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:01:45,401 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 18:01:45,401 Validation result (greedy) at epoch  26, step    63000: bleu:  25.55, loss: 44544.3047, ppl:   4.5027, duration: 18.6162s\n",
      "2020-02-12 18:01:55,586 Epoch  26 Step:    63100 Batch Loss:     1.724305 Tokens per Sec:    21599, Lr: 0.000300\n",
      "2020-02-12 18:02:05,558 Epoch  26 Step:    63200 Batch Loss:     1.697779 Tokens per Sec:    22488, Lr: 0.000300\n",
      "2020-02-12 18:02:15,444 Epoch  26 Step:    63300 Batch Loss:     1.665274 Tokens per Sec:    22424, Lr: 0.000300\n",
      "2020-02-12 18:02:25,332 Epoch  26 Step:    63400 Batch Loss:     1.396727 Tokens per Sec:    22748, Lr: 0.000300\n",
      "2020-02-12 18:02:35,187 Epoch  26 Step:    63500 Batch Loss:     1.763126 Tokens per Sec:    22528, Lr: 0.000300\n",
      "2020-02-12 18:02:45,029 Epoch  26 Step:    63600 Batch Loss:     1.371752 Tokens per Sec:    22915, Lr: 0.000300\n",
      "2020-02-12 18:02:54,919 Epoch  26 Step:    63700 Batch Loss:     1.700996 Tokens per Sec:    22140, Lr: 0.000300\n",
      "2020-02-12 18:03:04,748 Epoch  26 Step:    63800 Batch Loss:     1.579794 Tokens per Sec:    22391, Lr: 0.000300\n",
      "2020-02-12 18:03:14,602 Epoch  26 Step:    63900 Batch Loss:     1.330240 Tokens per Sec:    22577, Lr: 0.000300\n",
      "2020-02-12 18:03:17,395 Epoch  26: total training loss 3869.74\n",
      "2020-02-12 18:03:17,396 EPOCH 27\n",
      "2020-02-12 18:03:25,037 Epoch  27 Step:    64000 Batch Loss:     1.833237 Tokens per Sec:    20909, Lr: 0.000300\n",
      "2020-02-12 18:03:43,606 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:03:43,606 Saving new checkpoint.\n",
      "2020-02-12 18:03:43,794 Example #0\n",
      "2020-02-12 18:03:43,794 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:03:43,794 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:03:43,794 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima sha dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:03:43,794 Example #1\n",
      "2020-02-12 18:03:43,795 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:03:43,795 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:03:43,795 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 18:03:43,795 Example #2\n",
      "2020-02-12 18:03:43,795 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:03:43,795 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:03:43,795 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:03:43,795 Example #3\n",
      "2020-02-12 18:03:43,795 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:03:43,795 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:03:43,795 \tHypothesis: Ka kwagh u or nan lu eren sha gbenda u vough ga la .\n",
      "2020-02-12 18:03:43,795 Validation result (greedy) at epoch  27, step    64000: bleu:  25.26, loss: 44508.7266, ppl:   4.4973, duration: 18.7582s\n",
      "2020-02-12 18:03:53,801 Epoch  27 Step:    64100 Batch Loss:     1.582990 Tokens per Sec:    22825, Lr: 0.000300\n",
      "2020-02-12 18:04:03,704 Epoch  27 Step:    64200 Batch Loss:     1.733140 Tokens per Sec:    21853, Lr: 0.000300\n",
      "2020-02-12 18:04:13,667 Epoch  27 Step:    64300 Batch Loss:     1.572004 Tokens per Sec:    22485, Lr: 0.000300\n",
      "2020-02-12 18:04:23,619 Epoch  27 Step:    64400 Batch Loss:     1.562159 Tokens per Sec:    22166, Lr: 0.000300\n",
      "2020-02-12 18:04:33,576 Epoch  27 Step:    64500 Batch Loss:     1.349850 Tokens per Sec:    22343, Lr: 0.000300\n",
      "2020-02-12 18:04:43,441 Epoch  27 Step:    64600 Batch Loss:     1.555283 Tokens per Sec:    21489, Lr: 0.000300\n",
      "2020-02-12 18:04:53,432 Epoch  27 Step:    64700 Batch Loss:     1.464450 Tokens per Sec:    22416, Lr: 0.000300\n",
      "2020-02-12 18:05:03,349 Epoch  27 Step:    64800 Batch Loss:     1.556035 Tokens per Sec:    22205, Lr: 0.000300\n",
      "2020-02-12 18:05:13,244 Epoch  27 Step:    64900 Batch Loss:     1.659218 Tokens per Sec:    22365, Lr: 0.000300\n",
      "2020-02-12 18:05:23,181 Epoch  27 Step:    65000 Batch Loss:     1.738126 Tokens per Sec:    22776, Lr: 0.000300\n",
      "2020-02-12 18:05:41,858 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:05:41,858 Saving new checkpoint.\n",
      "2020-02-12 18:05:42,056 Example #0\n",
      "2020-02-12 18:05:42,056 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:05:42,057 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:05:42,057 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea eren kwagh u una wase ve u lun a dooshima shi lun a dooshima hen atô ve yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:05:42,057 Example #1\n",
      "2020-02-12 18:05:42,057 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:05:42,057 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:05:42,057 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon tsung je la ! — Rom .\n",
      "2020-02-12 18:05:42,057 Example #2\n",
      "2020-02-12 18:05:42,057 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:05:42,057 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:05:42,057 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:05:42,057 Example #3\n",
      "2020-02-12 18:05:42,058 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:05:42,058 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:05:42,058 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 18:05:42,058 Validation result (greedy) at epoch  27, step    65000: bleu:  25.36, loss: 44498.6055, ppl:   4.4957, duration: 18.8766s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 18:05:52,022 Epoch  27 Step:    65100 Batch Loss:     1.622702 Tokens per Sec:    21996, Lr: 0.000300\n",
      "2020-02-12 18:06:02,063 Epoch  27 Step:    65200 Batch Loss:     1.501931 Tokens per Sec:    22085, Lr: 0.000300\n",
      "2020-02-12 18:06:12,178 Epoch  27 Step:    65300 Batch Loss:     1.629987 Tokens per Sec:    22224, Lr: 0.000300\n",
      "2020-02-12 18:06:22,409 Epoch  27 Step:    65400 Batch Loss:     1.633025 Tokens per Sec:    21722, Lr: 0.000300\n",
      "2020-02-12 18:06:32,502 Epoch  27 Step:    65500 Batch Loss:     1.733178 Tokens per Sec:    21072, Lr: 0.000300\n",
      "2020-02-12 18:06:42,579 Epoch  27 Step:    65600 Batch Loss:     1.822144 Tokens per Sec:    21533, Lr: 0.000300\n",
      "2020-02-12 18:06:52,713 Epoch  27 Step:    65700 Batch Loss:     1.648252 Tokens per Sec:    21597, Lr: 0.000300\n",
      "2020-02-12 18:07:02,728 Epoch  27 Step:    65800 Batch Loss:     1.637809 Tokens per Sec:    22236, Lr: 0.000300\n",
      "2020-02-12 18:07:12,681 Epoch  27 Step:    65900 Batch Loss:     1.693966 Tokens per Sec:    21935, Lr: 0.000300\n",
      "2020-02-12 18:07:22,604 Epoch  27 Step:    66000 Batch Loss:     1.712889 Tokens per Sec:    22321, Lr: 0.000300\n",
      "2020-02-12 18:07:41,763 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:07:41,763 Saving new checkpoint.\n",
      "2020-02-12 18:07:41,951 Example #0\n",
      "2020-02-12 18:07:41,952 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:07:41,952 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:07:41,952 \tHypothesis: Er Yehova a ver ivese i tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi sha dooshima la , shi vea za hemen u lun a dooshima u a lu hen atô ve la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:07:41,952 Example #1\n",
      "2020-02-12 18:07:41,952 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:07:41,952 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:07:41,952 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 18:07:41,952 Example #2\n",
      "2020-02-12 18:07:41,952 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:07:41,952 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:07:41,953 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:07:41,953 Example #3\n",
      "2020-02-12 18:07:41,953 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:07:41,953 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:07:41,953 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 18:07:41,953 Validation result (greedy) at epoch  27, step    66000: bleu:  25.86, loss: 44215.5117, ppl:   4.4529, duration: 19.3480s\n",
      "2020-02-12 18:07:51,913 Epoch  27 Step:    66100 Batch Loss:     1.408067 Tokens per Sec:    22209, Lr: 0.000300\n",
      "2020-02-12 18:08:02,042 Epoch  27 Step:    66200 Batch Loss:     1.625484 Tokens per Sec:    21420, Lr: 0.000300\n",
      "2020-02-12 18:08:12,136 Epoch  27 Step:    66300 Batch Loss:     1.370028 Tokens per Sec:    21901, Lr: 0.000300\n",
      "2020-02-12 18:08:21,353 Epoch  27: total training loss 3854.13\n",
      "2020-02-12 18:08:21,354 EPOCH 28\n",
      "2020-02-12 18:08:22,544 Epoch  28 Step:    66400 Batch Loss:     1.539567 Tokens per Sec:    17759, Lr: 0.000300\n",
      "2020-02-12 18:08:32,613 Epoch  28 Step:    66500 Batch Loss:     1.510659 Tokens per Sec:    21864, Lr: 0.000300\n",
      "2020-02-12 18:08:42,684 Epoch  28 Step:    66600 Batch Loss:     1.654991 Tokens per Sec:    21632, Lr: 0.000300\n",
      "2020-02-12 18:08:52,839 Epoch  28 Step:    66700 Batch Loss:     1.649115 Tokens per Sec:    21274, Lr: 0.000300\n",
      "2020-02-12 18:09:02,882 Epoch  28 Step:    66800 Batch Loss:     1.389189 Tokens per Sec:    21843, Lr: 0.000300\n",
      "2020-02-12 18:09:12,920 Epoch  28 Step:    66900 Batch Loss:     1.546438 Tokens per Sec:    22272, Lr: 0.000300\n",
      "2020-02-12 18:09:23,115 Epoch  28 Step:    67000 Batch Loss:     1.760377 Tokens per Sec:    22011, Lr: 0.000300\n",
      "2020-02-12 18:09:42,858 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:09:42,858 Saving new checkpoint.\n",
      "2020-02-12 18:09:43,060 Example #0\n",
      "2020-02-12 18:09:43,061 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:09:43,061 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:09:43,061 \tHypothesis: Er Yehova a ver ivese i lu kwagh u tsan gbem sha won yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:09:43,061 Example #1\n",
      "2020-02-12 18:09:43,061 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:09:43,061 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:09:43,061 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u doon je la ! — Rom .\n",
      "2020-02-12 18:09:43,061 Example #2\n",
      "2020-02-12 18:09:43,061 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:09:43,061 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:09:43,062 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:09:43,062 Example #3\n",
      "2020-02-12 18:09:43,062 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:09:43,062 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:09:43,062 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 18:09:43,062 Validation result (greedy) at epoch  28, step    67000: bleu:  25.80, loss: 44074.8711, ppl:   4.4318, duration: 19.9463s\n",
      "2020-02-12 18:09:53,204 Epoch  28 Step:    67100 Batch Loss:     1.568015 Tokens per Sec:    21580, Lr: 0.000300\n",
      "2020-02-12 18:10:03,132 Epoch  28 Step:    67200 Batch Loss:     1.384739 Tokens per Sec:    22727, Lr: 0.000300\n",
      "2020-02-12 18:10:13,021 Epoch  28 Step:    67300 Batch Loss:     1.160034 Tokens per Sec:    21951, Lr: 0.000300\n",
      "2020-02-12 18:10:22,931 Epoch  28 Step:    67400 Batch Loss:     1.629328 Tokens per Sec:    22138, Lr: 0.000300\n",
      "2020-02-12 18:10:32,936 Epoch  28 Step:    67500 Batch Loss:     1.374674 Tokens per Sec:    22129, Lr: 0.000300\n",
      "2020-02-12 18:10:42,911 Epoch  28 Step:    67600 Batch Loss:     1.439200 Tokens per Sec:    22346, Lr: 0.000300\n",
      "2020-02-12 18:10:52,844 Epoch  28 Step:    67700 Batch Loss:     1.333939 Tokens per Sec:    22083, Lr: 0.000300\n",
      "2020-02-12 18:11:02,806 Epoch  28 Step:    67800 Batch Loss:     1.342506 Tokens per Sec:    21681, Lr: 0.000300\n",
      "2020-02-12 18:11:12,759 Epoch  28 Step:    67900 Batch Loss:     1.573518 Tokens per Sec:    22740, Lr: 0.000300\n",
      "2020-02-12 18:11:22,672 Epoch  28 Step:    68000 Batch Loss:     1.684764 Tokens per Sec:    22491, Lr: 0.000300\n",
      "2020-02-12 18:11:40,595 Example #0\n",
      "2020-02-12 18:11:40,596 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:11:40,596 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:11:40,596 \tHypothesis: Er Yehova a ver ivese i tsôron yô , gba u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve la , shi vea za hemen u lun ken dooshima u a lu hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:11:40,596 Example #1\n",
      "2020-02-12 18:11:40,596 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:11:40,596 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:11:40,596 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka sha kpôô yô , a lu ikyua a vese ! — Rom .\n",
      "2020-02-12 18:11:40,597 Example #2\n",
      "2020-02-12 18:11:40,597 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:11:40,597 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:11:40,597 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:11:40,597 Example #3\n",
      "2020-02-12 18:11:40,597 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:11:40,597 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:11:40,597 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 18:11:40,597 Validation result (greedy) at epoch  28, step    68000: bleu:  25.78, loss: 44164.0938, ppl:   4.4452, duration: 17.9249s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 18:11:50,625 Epoch  28 Step:    68100 Batch Loss:     1.531227 Tokens per Sec:    21840, Lr: 0.000300\n",
      "2020-02-12 18:12:00,501 Epoch  28 Step:    68200 Batch Loss:     1.345454 Tokens per Sec:    22238, Lr: 0.000300\n",
      "2020-02-12 18:12:10,423 Epoch  28 Step:    68300 Batch Loss:     1.617265 Tokens per Sec:    22208, Lr: 0.000300\n",
      "2020-02-12 18:12:20,465 Epoch  28 Step:    68400 Batch Loss:     1.731347 Tokens per Sec:    22151, Lr: 0.000300\n",
      "2020-02-12 18:12:30,397 Epoch  28 Step:    68500 Batch Loss:     1.498353 Tokens per Sec:    22634, Lr: 0.000300\n",
      "2020-02-12 18:12:40,264 Epoch  28 Step:    68600 Batch Loss:     1.360098 Tokens per Sec:    22784, Lr: 0.000300\n",
      "2020-02-12 18:12:50,163 Epoch  28 Step:    68700 Batch Loss:     1.640384 Tokens per Sec:    22473, Lr: 0.000300\n",
      "2020-02-12 18:13:00,026 Epoch  28 Step:    68800 Batch Loss:     1.543844 Tokens per Sec:    22450, Lr: 0.000300\n",
      "2020-02-12 18:13:04,790 Epoch  28: total training loss 3821.29\n",
      "2020-02-12 18:13:04,791 EPOCH 29\n",
      "2020-02-12 18:13:10,432 Epoch  29 Step:    68900 Batch Loss:     1.648355 Tokens per Sec:    20021, Lr: 0.000300\n",
      "2020-02-12 18:13:20,375 Epoch  29 Step:    69000 Batch Loss:     1.437651 Tokens per Sec:    22257, Lr: 0.000300\n",
      "2020-02-12 18:13:39,366 Example #0\n",
      "2020-02-12 18:13:39,366 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:13:39,366 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:13:39,366 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u gbem sha won yô , gba u noov man kasev vea eren kwagh sha dooshima ve u a lu hen atô ve la shi vea za hemen u lun a dooshima u a lu hen atô ve la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:13:39,366 Example #1\n",
      "2020-02-12 18:13:39,367 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:13:39,367 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:13:39,367 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka kwagh u a lu ikyua je la ! — Rom .\n",
      "2020-02-12 18:13:39,367 Example #2\n",
      "2020-02-12 18:13:39,367 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:13:39,367 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:13:39,367 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:13:39,367 Example #3\n",
      "2020-02-12 18:13:39,367 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:13:39,367 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:13:39,367 \tHypothesis: Alaghga a lu inja er or ngu eren kwagh sha inja ga nahan .\n",
      "2020-02-12 18:13:39,367 Validation result (greedy) at epoch  29, step    69000: bleu:  25.70, loss: 44289.9883, ppl:   4.4642, duration: 18.9915s\n",
      "2020-02-12 18:13:49,339 Epoch  29 Step:    69100 Batch Loss:     1.469833 Tokens per Sec:    21799, Lr: 0.000300\n",
      "2020-02-12 18:13:59,188 Epoch  29 Step:    69200 Batch Loss:     1.497443 Tokens per Sec:    22571, Lr: 0.000300\n",
      "2020-02-12 18:14:09,100 Epoch  29 Step:    69300 Batch Loss:     1.282275 Tokens per Sec:    21624, Lr: 0.000300\n",
      "2020-02-12 18:14:18,972 Epoch  29 Step:    69400 Batch Loss:     1.235603 Tokens per Sec:    22249, Lr: 0.000300\n",
      "2020-02-12 18:14:28,823 Epoch  29 Step:    69500 Batch Loss:     1.350500 Tokens per Sec:    22459, Lr: 0.000300\n",
      "2020-02-12 18:14:38,775 Epoch  29 Step:    69600 Batch Loss:     1.457592 Tokens per Sec:    22476, Lr: 0.000300\n",
      "2020-02-12 18:14:48,744 Epoch  29 Step:    69700 Batch Loss:     1.724861 Tokens per Sec:    22343, Lr: 0.000300\n",
      "2020-02-12 18:14:58,652 Epoch  29 Step:    69800 Batch Loss:     1.633235 Tokens per Sec:    22909, Lr: 0.000300\n",
      "2020-02-12 18:15:08,566 Epoch  29 Step:    69900 Batch Loss:     1.631249 Tokens per Sec:    22481, Lr: 0.000300\n",
      "2020-02-12 18:15:18,479 Epoch  29 Step:    70000 Batch Loss:     1.569043 Tokens per Sec:    22623, Lr: 0.000300\n",
      "2020-02-12 18:15:37,335 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:15:37,335 Saving new checkpoint.\n",
      "2020-02-12 18:15:37,521 Example #0\n",
      "2020-02-12 18:15:37,522 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:15:37,522 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:15:37,522 \tHypothesis: Er ivese i Yehova i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:15:37,522 Example #1\n",
      "2020-02-12 18:15:37,522 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:15:37,522 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:15:37,522 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u a mgbôghom je la ! — Rom .\n",
      "2020-02-12 18:15:37,522 Example #2\n",
      "2020-02-12 18:15:37,522 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:15:37,522 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:15:37,523 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:15:37,523 Example #3\n",
      "2020-02-12 18:15:37,523 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:15:37,523 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:15:37,523 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 18:15:37,523 Validation result (greedy) at epoch  29, step    70000: bleu:  25.51, loss: 44060.8906, ppl:   4.4297, duration: 19.0439s\n",
      "2020-02-12 18:15:47,448 Epoch  29 Step:    70100 Batch Loss:     1.834248 Tokens per Sec:    22449, Lr: 0.000300\n",
      "2020-02-12 18:15:57,462 Epoch  29 Step:    70200 Batch Loss:     1.374259 Tokens per Sec:    21797, Lr: 0.000300\n",
      "2020-02-12 18:16:07,303 Epoch  29 Step:    70300 Batch Loss:     1.645397 Tokens per Sec:    22438, Lr: 0.000300\n",
      "2020-02-12 18:16:17,184 Epoch  29 Step:    70400 Batch Loss:     1.347597 Tokens per Sec:    22111, Lr: 0.000300\n",
      "2020-02-12 18:16:27,083 Epoch  29 Step:    70500 Batch Loss:     1.479826 Tokens per Sec:    21891, Lr: 0.000300\n",
      "2020-02-12 18:16:37,026 Epoch  29 Step:    70600 Batch Loss:     1.441261 Tokens per Sec:    22206, Lr: 0.000300\n",
      "2020-02-12 18:16:47,016 Epoch  29 Step:    70700 Batch Loss:     1.719359 Tokens per Sec:    22357, Lr: 0.000300\n",
      "2020-02-12 18:16:56,973 Epoch  29 Step:    70800 Batch Loss:     1.678199 Tokens per Sec:    22320, Lr: 0.000300\n",
      "2020-02-12 18:17:06,949 Epoch  29 Step:    70900 Batch Loss:     1.493182 Tokens per Sec:    22012, Lr: 0.000300\n",
      "2020-02-12 18:17:16,960 Epoch  29 Step:    71000 Batch Loss:     1.365666 Tokens per Sec:    22477, Lr: 0.000300\n",
      "2020-02-12 18:17:35,785 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:17:35,786 Saving new checkpoint.\n",
      "2020-02-12 18:17:35,985 Example #0\n",
      "2020-02-12 18:17:35,985 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:17:35,985 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:17:35,985 \tHypothesis: Er ivese i lu kwagh u tsôron u Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea za hemen u lun a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:17:35,985 Example #1\n",
      "2020-02-12 18:17:35,986 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:17:35,986 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:17:35,986 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u kpilighyol ne , man ka shighe u a lu ikyua je la . — Rom .\n",
      "2020-02-12 18:17:35,986 Example #2\n",
      "2020-02-12 18:17:35,986 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:17:35,986 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:17:35,986 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 18:17:35,986 Example #3\n",
      "2020-02-12 18:17:35,986 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:17:35,986 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:17:35,986 \tHypothesis: Alaghga or a lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 18:17:35,987 Validation result (greedy) at epoch  29, step    71000: bleu:  25.96, loss: 43942.8281, ppl:   4.4121, duration: 19.0259s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 18:17:45,955 Epoch  29 Step:    71100 Batch Loss:     1.727092 Tokens per Sec:    22260, Lr: 0.000300\n",
      "2020-02-12 18:17:55,968 Epoch  29 Step:    71200 Batch Loss:     1.106232 Tokens per Sec:    22144, Lr: 0.000300\n",
      "2020-02-12 18:18:05,976 Epoch  29 Step:    71300 Batch Loss:     1.410406 Tokens per Sec:    22345, Lr: 0.000300\n",
      "2020-02-12 18:18:06,381 Epoch  29: total training loss 3800.10\n",
      "2020-02-12 18:18:06,381 EPOCH 30\n",
      "2020-02-12 18:18:16,267 Epoch  30 Step:    71400 Batch Loss:     1.575879 Tokens per Sec:    22000, Lr: 0.000300\n",
      "2020-02-12 18:18:26,311 Epoch  30 Step:    71500 Batch Loss:     1.388709 Tokens per Sec:    21468, Lr: 0.000300\n",
      "2020-02-12 18:18:36,239 Epoch  30 Step:    71600 Batch Loss:     1.634655 Tokens per Sec:    22447, Lr: 0.000300\n",
      "2020-02-12 18:18:46,171 Epoch  30 Step:    71700 Batch Loss:     1.814609 Tokens per Sec:    22184, Lr: 0.000300\n",
      "2020-02-12 18:18:56,108 Epoch  30 Step:    71800 Batch Loss:     1.648347 Tokens per Sec:    22093, Lr: 0.000300\n",
      "2020-02-12 18:19:06,113 Epoch  30 Step:    71900 Batch Loss:     1.598242 Tokens per Sec:    21608, Lr: 0.000300\n",
      "2020-02-12 18:19:16,270 Epoch  30 Step:    72000 Batch Loss:     1.660041 Tokens per Sec:    22040, Lr: 0.000300\n",
      "2020-02-12 18:19:35,913 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 18:19:35,913 Saving new checkpoint.\n",
      "2020-02-12 18:19:36,104 Example #0\n",
      "2020-02-12 18:19:36,104 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 18:19:36,104 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 18:19:36,104 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi sha kwagh u dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 18:19:36,104 Example #1\n",
      "2020-02-12 18:19:36,104 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 18:19:36,104 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 18:19:36,104 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka hen shighe la je la ! — Rom .\n",
      "2020-02-12 18:19:36,104 Example #2\n",
      "2020-02-12 18:19:36,105 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 18:19:36,105 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 18:19:36,105 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu .\n",
      "2020-02-12 18:19:36,105 Example #3\n",
      "2020-02-12 18:19:36,105 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 18:19:36,105 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 18:19:36,105 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 18:19:36,105 Validation result (greedy) at epoch  30, step    72000: bleu:  25.94, loss: 43766.0938, ppl:   4.3859, duration: 19.8351s\n",
      "2020-02-12 18:19:46,237 Epoch  30 Step:    72100 Batch Loss:     1.284048 Tokens per Sec:    21788, Lr: 0.000300\n",
      "2020-02-12 20:38:06,989 Epoch  58 Step:   142000 Batch Loss:     1.649239 Tokens per Sec:    21882, Lr: 0.000300\n",
      "2020-02-12 20:38:26,303 Example #0\n",
      "2020-02-12 20:38:26,304 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:38:26,304 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:38:26,304 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo sha er dooshima ve una seer taver yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:38:26,304 Example #1\n",
      "2020-02-12 20:38:26,304 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:38:26,304 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:38:26,304 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:38:26,304 Example #2\n",
      "2020-02-12 20:38:26,304 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:38:26,305 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:38:26,305 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:38:26,305 Example #3\n",
      "2020-02-12 20:38:26,305 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:38:26,305 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:38:26,305 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:38:26,305 Validation result (greedy) at epoch  58, step   142000: bleu:  28.14, loss: 41258.6484, ppl:   4.0297, duration: 19.3153s\n",
      "2020-02-12 20:38:36,396 Epoch  58 Step:   142100 Batch Loss:     1.189728 Tokens per Sec:    21759, Lr: 0.000300\n",
      "2020-02-12 20:38:46,575 Epoch  58 Step:   142200 Batch Loss:     1.555159 Tokens per Sec:    21747, Lr: 0.000300\n",
      "2020-02-12 20:38:56,673 Epoch  58 Step:   142300 Batch Loss:     1.462615 Tokens per Sec:    22033, Lr: 0.000300\n",
      "2020-02-12 20:39:06,679 Epoch  58 Step:   142400 Batch Loss:     1.330904 Tokens per Sec:    22019, Lr: 0.000300\n",
      "2020-02-12 20:39:16,673 Epoch  58 Step:   142500 Batch Loss:     1.320575 Tokens per Sec:    22309, Lr: 0.000300\n",
      "2020-02-12 20:39:26,777 Epoch  58 Step:   142600 Batch Loss:     1.345322 Tokens per Sec:    22151, Lr: 0.000300\n",
      "2020-02-12 20:39:28,365 Epoch  58: total training loss 3443.92\n",
      "2020-02-12 20:39:28,365 EPOCH 59\n",
      "2020-02-12 20:39:37,128 Epoch  59 Step:   142700 Batch Loss:     1.680768 Tokens per Sec:    21219, Lr: 0.000300\n",
      "2020-02-12 20:39:47,103 Epoch  59 Step:   142800 Batch Loss:     1.409805 Tokens per Sec:    21886, Lr: 0.000300\n",
      "2020-02-12 20:39:57,097 Epoch  59 Step:   142900 Batch Loss:     1.261157 Tokens per Sec:    22112, Lr: 0.000300\n",
      "2020-02-12 20:40:07,101 Epoch  59 Step:   143000 Batch Loss:     1.508628 Tokens per Sec:    22201, Lr: 0.000300\n",
      "2020-02-12 20:40:25,593 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 20:40:25,593 Saving new checkpoint.\n",
      "2020-02-12 20:40:25,782 Example #0\n",
      "2020-02-12 20:40:25,782 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:40:25,782 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:40:25,782 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem sha won yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u mimi shi vea za hemen u lun a dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:40:25,782 Example #1\n",
      "2020-02-12 20:40:25,783 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:40:25,783 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:40:25,783 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:40:25,783 Example #2\n",
      "2020-02-12 20:40:25,783 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:40:25,783 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:40:25,783 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:40:25,783 Example #3\n",
      "2020-02-12 20:40:25,783 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:40:25,784 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:40:25,784 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:40:25,784 Validation result (greedy) at epoch  59, step   143000: bleu:  28.19, loss: 41192.2695, ppl:   4.0206, duration: 18.6818s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 20:40:35,739 Epoch  59 Step:   143100 Batch Loss:     1.655330 Tokens per Sec:    21997, Lr: 0.000300\n",
      "2020-02-12 20:40:45,861 Epoch  59 Step:   143200 Batch Loss:     1.374167 Tokens per Sec:    21439, Lr: 0.000300\n",
      "2020-02-12 20:40:55,964 Epoch  59 Step:   143300 Batch Loss:     1.405934 Tokens per Sec:    22054, Lr: 0.000300\n",
      "2020-02-12 20:41:05,941 Epoch  59 Step:   143400 Batch Loss:     1.371737 Tokens per Sec:    22278, Lr: 0.000300\n",
      "2020-02-12 20:41:15,827 Epoch  59 Step:   143500 Batch Loss:     1.304805 Tokens per Sec:    22822, Lr: 0.000300\n",
      "2020-02-12 20:41:25,682 Epoch  59 Step:   143600 Batch Loss:     1.395527 Tokens per Sec:    22346, Lr: 0.000300\n",
      "2020-02-12 20:41:35,595 Epoch  59 Step:   143700 Batch Loss:     1.518479 Tokens per Sec:    22494, Lr: 0.000300\n",
      "2020-02-12 20:41:45,622 Epoch  59 Step:   143800 Batch Loss:     1.373104 Tokens per Sec:    21925, Lr: 0.000300\n",
      "2020-02-12 20:41:55,535 Epoch  59 Step:   143900 Batch Loss:     1.451918 Tokens per Sec:    22685, Lr: 0.000300\n",
      "2020-02-12 20:42:05,464 Epoch  59 Step:   144000 Batch Loss:     1.513248 Tokens per Sec:    22161, Lr: 0.000300\n",
      "2020-02-12 20:42:24,346 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 20:42:24,346 Saving new checkpoint.\n",
      "2020-02-12 20:42:24,544 Example #0\n",
      "2020-02-12 20:42:24,545 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:42:24,545 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:42:24,545 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve la , shi vea za hemen u lun a dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:42:24,545 Example #1\n",
      "2020-02-12 20:42:24,545 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:42:24,545 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:42:24,545 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:42:24,545 Example #2\n",
      "2020-02-12 20:42:24,546 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:42:24,546 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:42:24,546 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:42:24,546 Example #3\n",
      "2020-02-12 20:42:24,546 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:42:24,546 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:42:24,546 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:42:24,546 Validation result (greedy) at epoch  59, step   144000: bleu:  28.03, loss: 41174.0195, ppl:   4.0182, duration: 19.0813s\n",
      "2020-02-12 20:42:34,504 Epoch  59 Step:   144100 Batch Loss:     1.442042 Tokens per Sec:    22134, Lr: 0.000300\n",
      "2020-02-12 20:42:44,452 Epoch  59 Step:   144200 Batch Loss:     1.586396 Tokens per Sec:    22378, Lr: 0.000300\n",
      "2020-02-12 20:42:54,387 Epoch  59 Step:   144300 Batch Loss:     1.750405 Tokens per Sec:    21873, Lr: 0.000300\n",
      "2020-02-12 20:43:04,335 Epoch  59 Step:   144400 Batch Loss:     1.335457 Tokens per Sec:    22193, Lr: 0.000300\n",
      "2020-02-12 20:43:14,285 Epoch  59 Step:   144500 Batch Loss:     1.366822 Tokens per Sec:    22546, Lr: 0.000300\n",
      "2020-02-12 20:43:24,223 Epoch  59 Step:   144600 Batch Loss:     1.430219 Tokens per Sec:    22720, Lr: 0.000300\n",
      "2020-02-12 20:43:34,267 Epoch  59 Step:   144700 Batch Loss:     1.255848 Tokens per Sec:    22050, Lr: 0.000300\n",
      "2020-02-12 20:43:44,418 Epoch  59 Step:   144800 Batch Loss:     1.433099 Tokens per Sec:    22049, Lr: 0.000300\n",
      "2020-02-12 20:43:54,633 Epoch  59 Step:   144900 Batch Loss:     1.580696 Tokens per Sec:    22078, Lr: 0.000300\n",
      "2020-02-12 20:44:04,776 Epoch  59 Step:   145000 Batch Loss:     1.564681 Tokens per Sec:    21270, Lr: 0.000300\n",
      "2020-02-12 20:44:24,708 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 20:44:24,709 Saving new checkpoint.\n",
      "2020-02-12 20:44:24,899 Example #0\n",
      "2020-02-12 20:44:24,899 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:44:24,899 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:44:24,900 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u una fatyô u vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:44:24,900 Example #1\n",
      "2020-02-12 20:44:24,900 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:44:24,900 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:44:24,900 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:44:24,900 Example #2\n",
      "2020-02-12 20:44:24,900 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:44:24,900 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:44:24,900 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:44:24,901 Example #3\n",
      "2020-02-12 20:44:24,901 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:44:24,901 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:44:24,901 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:44:24,901 Validation result (greedy) at epoch  59, step   145000: bleu:  28.48, loss: 41079.0156, ppl:   4.0053, duration: 20.1248s\n",
      "2020-02-12 20:44:31,869 Epoch  59: total training loss 3429.87\n",
      "2020-02-12 20:44:31,870 EPOCH 60\n",
      "2020-02-12 20:44:35,280 Epoch  60 Step:   145100 Batch Loss:     1.214325 Tokens per Sec:    20652, Lr: 0.000300\n",
      "2020-02-12 20:44:45,321 Epoch  60 Step:   145200 Batch Loss:     1.116316 Tokens per Sec:    21821, Lr: 0.000300\n",
      "2020-02-12 20:44:55,212 Epoch  60 Step:   145300 Batch Loss:     1.525003 Tokens per Sec:    22632, Lr: 0.000300\n",
      "2020-02-12 20:45:05,130 Epoch  60 Step:   145400 Batch Loss:     1.480217 Tokens per Sec:    22628, Lr: 0.000300\n",
      "2020-02-12 20:45:15,176 Epoch  60 Step:   145500 Batch Loss:     1.371782 Tokens per Sec:    21804, Lr: 0.000300\n",
      "2020-02-12 20:45:25,329 Epoch  60 Step:   145600 Batch Loss:     1.311836 Tokens per Sec:    21782, Lr: 0.000300\n",
      "2020-02-12 20:45:35,380 Epoch  60 Step:   145700 Batch Loss:     1.487752 Tokens per Sec:    21159, Lr: 0.000300\n",
      "2020-02-12 20:45:45,438 Epoch  60 Step:   145800 Batch Loss:     1.526056 Tokens per Sec:    22168, Lr: 0.000300\n",
      "2020-02-12 20:45:55,487 Epoch  60 Step:   145900 Batch Loss:     1.605019 Tokens per Sec:    21551, Lr: 0.000300\n",
      "2020-02-12 20:46:05,477 Epoch  60 Step:   146000 Batch Loss:     1.284405 Tokens per Sec:    21967, Lr: 0.000300\n",
      "2020-02-12 20:46:24,191 Example #0\n",
      "2020-02-12 20:46:24,191 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:46:24,191 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:46:24,192 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo sha er dooshima ve una seer taver yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:46:24,192 Example #1\n",
      "2020-02-12 20:46:24,192 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:46:24,192 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:46:24,192 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man ka shighe u a lu van la je la ! — Rom .\n",
      "2020-02-12 20:46:24,192 Example #2\n",
      "2020-02-12 20:46:24,192 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:46:24,192 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:46:24,192 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:46:24,192 Example #3\n",
      "2020-02-12 20:46:24,193 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:46:24,193 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:46:24,193 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:46:24,193 Validation result (greedy) at epoch  60, step   146000: bleu:  28.29, loss: 41082.6055, ppl:   4.0058, duration: 18.7152s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 20:46:34,109 Epoch  60 Step:   146100 Batch Loss:     1.409818 Tokens per Sec:    22663, Lr: 0.000300\n",
      "2020-02-12 20:46:43,994 Epoch  60 Step:   146200 Batch Loss:     1.582477 Tokens per Sec:    22008, Lr: 0.000300\n",
      "2020-02-12 20:46:54,001 Epoch  60 Step:   146300 Batch Loss:     1.423970 Tokens per Sec:    21738, Lr: 0.000300\n",
      "2020-02-12 20:47:04,026 Epoch  60 Step:   146400 Batch Loss:     1.432156 Tokens per Sec:    22273, Lr: 0.000300\n",
      "2020-02-12 20:47:14,184 Epoch  60 Step:   146500 Batch Loss:     1.406879 Tokens per Sec:    21858, Lr: 0.000300\n",
      "2020-02-12 20:47:24,299 Epoch  60 Step:   146600 Batch Loss:     1.477002 Tokens per Sec:    21966, Lr: 0.000300\n",
      "2020-02-12 20:47:34,425 Epoch  60 Step:   146700 Batch Loss:     1.466704 Tokens per Sec:    22221, Lr: 0.000300\n",
      "2020-02-12 20:47:44,529 Epoch  60 Step:   146800 Batch Loss:     1.531728 Tokens per Sec:    22174, Lr: 0.000300\n",
      "2020-02-12 20:47:54,756 Epoch  60 Step:   146900 Batch Loss:     1.330328 Tokens per Sec:    21745, Lr: 0.000300\n",
      "2020-02-12 20:48:04,848 Epoch  60 Step:   147000 Batch Loss:     1.371225 Tokens per Sec:    21815, Lr: 0.000300\n",
      "2020-02-12 20:48:23,007 Example #0\n",
      "2020-02-12 20:48:23,008 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:48:23,008 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:48:23,008 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo sha er dooshima ve una seer taver yô . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:48:23,008 Example #1\n",
      "2020-02-12 20:48:23,008 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:48:23,008 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:48:23,008 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man ka shighe u a mgbôghom je la ! — Rom .\n",
      "2020-02-12 20:48:23,008 Example #2\n",
      "2020-02-12 20:48:23,008 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:48:23,008 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:48:23,008 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:48:23,008 Example #3\n",
      "2020-02-12 20:48:23,009 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:48:23,009 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:48:23,009 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:48:23,009 Validation result (greedy) at epoch  60, step   147000: bleu:  27.90, loss: 41109.3867, ppl:   4.0094, duration: 18.1600s\n",
      "2020-02-12 20:48:32,849 Epoch  60 Step:   147100 Batch Loss:     1.518140 Tokens per Sec:    22258, Lr: 0.000300\n",
      "2020-02-12 20:48:42,700 Epoch  60 Step:   147200 Batch Loss:     1.556091 Tokens per Sec:    22913, Lr: 0.000300\n",
      "2020-02-12 20:48:52,647 Epoch  60 Step:   147300 Batch Loss:     1.265665 Tokens per Sec:    22190, Lr: 0.000300\n",
      "2020-02-12 20:49:02,571 Epoch  60 Step:   147400 Batch Loss:     1.290607 Tokens per Sec:    22142, Lr: 0.000300\n",
      "2020-02-12 20:49:12,409 Epoch  60 Step:   147500 Batch Loss:     1.525233 Tokens per Sec:    22600, Lr: 0.000300\n",
      "2020-02-12 20:49:14,961 Epoch  60: total training loss 3426.66\n",
      "2020-02-12 20:49:14,961 EPOCH 61\n",
      "2020-02-12 20:49:22,504 Epoch  61 Step:   147600 Batch Loss:     1.378971 Tokens per Sec:    21721, Lr: 0.000300\n",
      "2020-02-12 20:49:32,359 Epoch  61 Step:   147700 Batch Loss:     1.672227 Tokens per Sec:    22312, Lr: 0.000300\n",
      "2020-02-12 20:49:42,227 Epoch  61 Step:   147800 Batch Loss:     1.321020 Tokens per Sec:    22710, Lr: 0.000300\n",
      "2020-02-12 20:49:52,150 Epoch  61 Step:   147900 Batch Loss:     1.464232 Tokens per Sec:    22211, Lr: 0.000300\n",
      "2020-02-12 20:50:02,302 Epoch  61 Step:   148000 Batch Loss:     1.264526 Tokens per Sec:    21572, Lr: 0.000300\n",
      "2020-02-12 20:50:22,504 Example #0\n",
      "2020-02-12 20:50:22,505 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:50:22,505 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:50:22,505 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu hen atô ve la shi vea lu a dooshima u a lu a mi la her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:50:22,505 Example #1\n",
      "2020-02-12 20:50:22,505 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:50:22,505 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:50:22,505 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a imba shighe u doon tsung ne , man ka shighe u a lu van la je la ! — Rom .\n",
      "2020-02-12 20:50:22,505 Example #2\n",
      "2020-02-12 20:50:22,505 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:50:22,505 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:50:22,506 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:50:22,506 Example #3\n",
      "2020-02-12 20:50:22,506 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:50:22,506 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:50:22,506 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 20:50:22,506 Validation result (greedy) at epoch  61, step   148000: bleu:  27.85, loss: 41156.0664, ppl:   4.0157, duration: 20.2031s\n",
      "2020-02-12 20:50:32,685 Epoch  61 Step:   148100 Batch Loss:     1.448269 Tokens per Sec:    21696, Lr: 0.000300\n",
      "2020-02-12 20:50:42,906 Epoch  61 Step:   148200 Batch Loss:     1.584460 Tokens per Sec:    22074, Lr: 0.000300\n",
      "2020-02-12 20:50:53,074 Epoch  61 Step:   148300 Batch Loss:     1.728064 Tokens per Sec:    22181, Lr: 0.000300\n",
      "2020-02-12 20:51:03,161 Epoch  61 Step:   148400 Batch Loss:     1.250297 Tokens per Sec:    21566, Lr: 0.000300\n",
      "2020-02-12 20:51:13,267 Epoch  61 Step:   148500 Batch Loss:     1.240981 Tokens per Sec:    21921, Lr: 0.000300\n",
      "2020-02-12 20:51:23,416 Epoch  61 Step:   148600 Batch Loss:     1.278483 Tokens per Sec:    21912, Lr: 0.000300\n",
      "2020-02-12 20:51:33,326 Epoch  61 Step:   148700 Batch Loss:     1.544158 Tokens per Sec:    22556, Lr: 0.000300\n",
      "2020-02-12 20:51:43,220 Epoch  61 Step:   148800 Batch Loss:     1.419829 Tokens per Sec:    22130, Lr: 0.000300\n",
      "2020-02-12 20:51:53,168 Epoch  61 Step:   148900 Batch Loss:     1.257829 Tokens per Sec:    22179, Lr: 0.000300\n",
      "2020-02-12 20:52:03,094 Epoch  61 Step:   149000 Batch Loss:     1.153634 Tokens per Sec:    21443, Lr: 0.000300\n",
      "2020-02-12 20:52:21,575 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 20:52:21,576 Saving new checkpoint.\n",
      "2020-02-12 20:52:21,773 Example #0\n",
      "2020-02-12 20:52:21,773 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:52:21,773 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:52:21,773 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu hen atô ve la shi vea lu a dooshima u a lu a mi la her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:52:21,774 Example #1\n",
      "2020-02-12 20:52:21,774 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:52:21,774 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:52:21,774 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man a mgbôghom ! — Rom .\n",
      "2020-02-12 20:52:21,774 Example #2\n",
      "2020-02-12 20:52:21,774 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:52:21,774 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:52:21,774 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:52:21,774 Example #3\n",
      "2020-02-12 20:52:21,774 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:52:21,775 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:52:21,775 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:52:21,775 Validation result (greedy) at epoch  61, step   149000: bleu:  28.14, loss: 40998.1211, ppl:   3.9944, duration: 18.6799s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 20:52:31,621 Epoch  61 Step:   149100 Batch Loss:     1.441156 Tokens per Sec:    22866, Lr: 0.000300\n",
      "2020-02-12 20:52:41,491 Epoch  61 Step:   149200 Batch Loss:     1.400879 Tokens per Sec:    22117, Lr: 0.000300\n",
      "2020-02-12 20:52:51,454 Epoch  61 Step:   149300 Batch Loss:     1.300657 Tokens per Sec:    22400, Lr: 0.000300\n",
      "2020-02-12 20:53:01,391 Epoch  61 Step:   149400 Batch Loss:     1.552772 Tokens per Sec:    21856, Lr: 0.000300\n",
      "2020-02-12 20:53:11,395 Epoch  61 Step:   149500 Batch Loss:     1.522980 Tokens per Sec:    21900, Lr: 0.000300\n",
      "2020-02-12 20:53:21,480 Epoch  61 Step:   149600 Batch Loss:     1.515286 Tokens per Sec:    21542, Lr: 0.000300\n",
      "2020-02-12 20:53:31,543 Epoch  61 Step:   149700 Batch Loss:     1.347546 Tokens per Sec:    22254, Lr: 0.000300\n",
      "2020-02-12 20:53:41,618 Epoch  61 Step:   149800 Batch Loss:     1.403254 Tokens per Sec:    22236, Lr: 0.000300\n",
      "2020-02-12 20:53:51,674 Epoch  61 Step:   149900 Batch Loss:     1.528272 Tokens per Sec:    21792, Lr: 0.000300\n",
      "2020-02-12 20:54:00,293 Epoch  61: total training loss 3427.88\n",
      "2020-02-12 20:54:00,294 EPOCH 62\n",
      "2020-02-12 20:54:01,994 Epoch  62 Step:   150000 Batch Loss:     1.459719 Tokens per Sec:    18242, Lr: 0.000300\n",
      "2020-02-12 20:54:20,810 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 20:54:20,811 Saving new checkpoint.\n",
      "2020-02-12 20:54:20,997 Example #0\n",
      "2020-02-12 20:54:20,997 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:54:20,997 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:54:20,998 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u vesen yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:54:20,998 Example #1\n",
      "2020-02-12 20:54:20,998 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:54:20,998 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:54:20,998 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:54:20,998 Example #2\n",
      "2020-02-12 20:54:20,998 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:54:20,998 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:54:20,998 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:54:20,998 Example #3\n",
      "2020-02-12 20:54:20,998 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:54:20,999 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:54:20,999 \tHypothesis: Alaghga or nana lu eren kwagh u vough ga .\n",
      "2020-02-12 20:54:20,999 Validation result (greedy) at epoch  62, step   150000: bleu:  27.91, loss: 40995.7305, ppl:   3.9940, duration: 19.0037s\n",
      "2020-02-12 20:54:30,856 Epoch  62 Step:   150100 Batch Loss:     1.141374 Tokens per Sec:    22198, Lr: 0.000300\n",
      "2020-02-12 20:54:40,720 Epoch  62 Step:   150200 Batch Loss:     1.360491 Tokens per Sec:    22391, Lr: 0.000300\n",
      "2020-02-12 20:54:50,792 Epoch  62 Step:   150300 Batch Loss:     1.500698 Tokens per Sec:    21524, Lr: 0.000300\n",
      "2020-02-12 20:55:00,955 Epoch  62 Step:   150400 Batch Loss:     1.158732 Tokens per Sec:    22040, Lr: 0.000300\n",
      "2020-02-12 20:55:11,097 Epoch  62 Step:   150500 Batch Loss:     1.405347 Tokens per Sec:    21578, Lr: 0.000300\n",
      "2020-02-12 20:55:21,243 Epoch  62 Step:   150600 Batch Loss:     1.268173 Tokens per Sec:    21782, Lr: 0.000300\n",
      "2020-02-12 20:55:31,396 Epoch  62 Step:   150700 Batch Loss:     1.367640 Tokens per Sec:    21504, Lr: 0.000300\n",
      "2020-02-12 20:55:41,577 Epoch  62 Step:   150800 Batch Loss:     1.221848 Tokens per Sec:    21900, Lr: 0.000300\n",
      "2020-02-12 20:55:51,705 Epoch  62 Step:   150900 Batch Loss:     1.442880 Tokens per Sec:    22148, Lr: 0.000300\n",
      "2020-02-12 20:56:01,678 Epoch  62 Step:   151000 Batch Loss:     1.450268 Tokens per Sec:    21682, Lr: 0.000300\n",
      "2020-02-12 20:56:20,320 Example #0\n",
      "2020-02-12 20:56:20,320 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:56:20,320 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:56:20,320 \tHypothesis: Er Yehova a ver ivese i tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve la shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:56:20,320 Example #1\n",
      "2020-02-12 20:56:20,320 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:56:20,320 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:56:20,320 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u kpilighyol ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:56:20,320 Example #2\n",
      "2020-02-12 20:56:20,321 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:56:20,321 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:56:20,321 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:56:20,321 Example #3\n",
      "2020-02-12 20:56:20,321 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:56:20,321 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:56:20,321 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 20:56:20,321 Validation result (greedy) at epoch  62, step   151000: bleu:  28.08, loss: 41003.5430, ppl:   3.9951, duration: 18.6427s\n",
      "2020-02-12 20:56:30,227 Epoch  62 Step:   151100 Batch Loss:     1.490563 Tokens per Sec:    22839, Lr: 0.000300\n",
      "2020-02-12 20:56:40,100 Epoch  62 Step:   151200 Batch Loss:     1.310824 Tokens per Sec:    22447, Lr: 0.000300\n",
      "2020-02-12 20:56:50,056 Epoch  62 Step:   151300 Batch Loss:     1.531437 Tokens per Sec:    22189, Lr: 0.000300\n",
      "2020-02-12 20:56:59,952 Epoch  62 Step:   151400 Batch Loss:     1.529835 Tokens per Sec:    22204, Lr: 0.000300\n",
      "2020-02-12 20:57:09,894 Epoch  62 Step:   151500 Batch Loss:     1.161585 Tokens per Sec:    22171, Lr: 0.000300\n",
      "2020-02-12 20:57:19,922 Epoch  62 Step:   151600 Batch Loss:     1.279486 Tokens per Sec:    22087, Lr: 0.000300\n",
      "2020-02-12 20:57:29,814 Epoch  62 Step:   151700 Batch Loss:     1.544322 Tokens per Sec:    22731, Lr: 0.000300\n",
      "2020-02-12 20:57:39,724 Epoch  62 Step:   151800 Batch Loss:     1.487324 Tokens per Sec:    22343, Lr: 0.000300\n",
      "2020-02-12 20:57:49,674 Epoch  62 Step:   151900 Batch Loss:     1.487652 Tokens per Sec:    22125, Lr: 0.000300\n",
      "2020-02-12 20:57:59,549 Epoch  62 Step:   152000 Batch Loss:     1.435196 Tokens per Sec:    22413, Lr: 0.000300\n",
      "2020-02-12 20:58:18,101 Example #0\n",
      "2020-02-12 20:58:18,101 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 20:58:18,101 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 20:58:18,101 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem sha won yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea lu a dooshima u a lu vesen la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 20:58:18,101 Example #1\n",
      "2020-02-12 20:58:18,101 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 20:58:18,102 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 20:58:18,102 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 20:58:18,102 Example #2\n",
      "2020-02-12 20:58:18,102 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 20:58:18,102 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 20:58:18,102 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken gar u Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 20:58:18,102 Example #3\n",
      "2020-02-12 20:58:18,102 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 20:58:18,102 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 20:58:18,102 \tHypothesis: Alaghga or nana lu eren kwagh u vough ga .\n",
      "2020-02-12 20:58:18,102 Validation result (greedy) at epoch  62, step   152000: bleu:  27.93, loss: 40997.4297, ppl:   3.9943, duration: 18.5529s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 20:58:28,091 Epoch  62 Step:   152100 Batch Loss:     1.605337 Tokens per Sec:    22337, Lr: 0.000300\n",
      "2020-02-12 20:58:38,061 Epoch  62 Step:   152200 Batch Loss:     1.457781 Tokens per Sec:    22304, Lr: 0.000300\n",
      "2020-02-12 20:58:48,192 Epoch  62 Step:   152300 Batch Loss:     1.252030 Tokens per Sec:    21782, Lr: 0.000300\n",
      "2020-02-12 20:58:58,216 Epoch  62 Step:   152400 Batch Loss:     1.487466 Tokens per Sec:    22369, Lr: 0.000300\n",
      "2020-02-12 20:59:02,297 Epoch  62: total training loss 3407.62\n",
      "2020-02-12 20:59:02,298 EPOCH 63\n",
      "2020-02-12 20:59:08,413 Epoch  63 Step:   152500 Batch Loss:     0.917715 Tokens per Sec:    20793, Lr: 0.000300\n",
      "2020-02-12 20:59:18,263 Epoch  63 Step:   152600 Batch Loss:     1.154300 Tokens per Sec:    22631, Lr: 0.000300\n",
      "2020-02-12 20:59:28,117 Epoch  63 Step:   152700 Batch Loss:     0.896696 Tokens per Sec:    21794, Lr: 0.000300\n",
      "2020-02-12 20:59:37,994 Epoch  63 Step:   152800 Batch Loss:     1.533581 Tokens per Sec:    22516, Lr: 0.000300\n",
      "2020-02-12 20:59:47,914 Epoch  63 Step:   152900 Batch Loss:     1.351405 Tokens per Sec:    22496, Lr: 0.000300\n",
      "2020-02-12 20:59:57,949 Epoch  63 Step:   153000 Batch Loss:     1.559299 Tokens per Sec:    21684, Lr: 0.000300\n",
      "2020-02-12 21:00:16,846 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:00:16,847 Saving new checkpoint.\n",
      "2020-02-12 21:00:17,047 Example #0\n",
      "2020-02-12 21:00:17,048 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:00:17,048 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:00:17,048 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea lu a dooshima u a lu a mi la her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:00:17,048 Example #1\n",
      "2020-02-12 21:00:17,048 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:00:17,048 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:00:17,048 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:00:17,048 Example #2\n",
      "2020-02-12 21:00:17,049 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:00:17,049 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:00:17,049 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:00:17,049 Example #3\n",
      "2020-02-12 21:00:17,049 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:00:17,049 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:00:17,049 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:00:17,049 Validation result (greedy) at epoch  63, step   153000: bleu:  28.02, loss: 40971.5273, ppl:   3.9908, duration: 19.1001s\n",
      "2020-02-12 21:00:27,332 Epoch  63 Step:   153100 Batch Loss:     1.392670 Tokens per Sec:    21835, Lr: 0.000300\n",
      "2020-02-12 21:00:37,412 Epoch  63 Step:   153200 Batch Loss:     1.463651 Tokens per Sec:    22296, Lr: 0.000300\n",
      "2020-02-12 21:00:47,501 Epoch  63 Step:   153300 Batch Loss:     1.382484 Tokens per Sec:    22122, Lr: 0.000300\n",
      "2020-02-12 21:00:57,605 Epoch  63 Step:   153400 Batch Loss:     1.327636 Tokens per Sec:    21560, Lr: 0.000300\n",
      "2020-02-12 21:01:07,735 Epoch  63 Step:   153500 Batch Loss:     1.469077 Tokens per Sec:    22450, Lr: 0.000300\n",
      "2020-02-12 21:01:17,799 Epoch  63 Step:   153600 Batch Loss:     1.399526 Tokens per Sec:    21692, Lr: 0.000300\n",
      "2020-02-12 21:01:27,812 Epoch  63 Step:   153700 Batch Loss:     1.211877 Tokens per Sec:    22177, Lr: 0.000300\n",
      "2020-02-12 21:01:37,692 Epoch  63 Step:   153800 Batch Loss:     1.342598 Tokens per Sec:    22431, Lr: 0.000300\n",
      "2020-02-12 21:01:47,640 Epoch  63 Step:   153900 Batch Loss:     1.259464 Tokens per Sec:    22478, Lr: 0.000300\n",
      "2020-02-12 21:01:57,534 Epoch  63 Step:   154000 Batch Loss:     1.611214 Tokens per Sec:    22478, Lr: 0.000300\n",
      "2020-02-12 21:02:16,004 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:02:16,004 Saving new checkpoint.\n",
      "2020-02-12 21:02:16,196 Example #0\n",
      "2020-02-12 21:02:16,196 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:02:16,196 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:02:16,196 \tHypothesis: Er mserakaa u Yehova a ver la a lu kwagh u tsan gbem yô , gba hange hange u noov man kasev vea nôngo tsung vea lu a dooshima ve la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:02:16,197 Example #1\n",
      "2020-02-12 21:02:16,197 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:02:16,197 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:02:16,197 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u kpilighyol ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:02:16,197 Example #2\n",
      "2020-02-12 21:02:16,197 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:02:16,197 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:02:16,197 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:02:16,197 Example #3\n",
      "2020-02-12 21:02:16,197 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:02:16,198 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:02:16,198 \tHypothesis: Alaghga or nana lu eren kwagh u vough ga .\n",
      "2020-02-12 21:02:16,198 Validation result (greedy) at epoch  63, step   154000: bleu:  28.04, loss: 40796.3867, ppl:   3.9672, duration: 18.6636s\n",
      "2020-02-12 21:02:26,105 Epoch  63 Step:   154100 Batch Loss:     1.318587 Tokens per Sec:    22005, Lr: 0.000300\n",
      "2020-02-12 21:02:36,261 Epoch  63 Step:   154200 Batch Loss:     1.436596 Tokens per Sec:    21237, Lr: 0.000300\n",
      "2020-02-12 21:02:46,541 Epoch  63 Step:   154300 Batch Loss:     1.437757 Tokens per Sec:    21601, Lr: 0.000300\n",
      "2020-02-12 21:02:56,775 Epoch  63 Step:   154400 Batch Loss:     1.295896 Tokens per Sec:    21555, Lr: 0.000300\n",
      "2020-02-12 21:03:06,958 Epoch  63 Step:   154500 Batch Loss:     1.450823 Tokens per Sec:    21843, Lr: 0.000300\n",
      "2020-02-12 21:03:16,960 Epoch  63 Step:   154600 Batch Loss:     1.444382 Tokens per Sec:    21928, Lr: 0.000300\n",
      "2020-02-12 21:03:27,276 Epoch  63 Step:   154700 Batch Loss:     1.487446 Tokens per Sec:    21794, Lr: 0.000300\n",
      "2020-02-12 21:03:37,425 Epoch  63 Step:   154800 Batch Loss:     1.332296 Tokens per Sec:    21668, Lr: 0.000300\n",
      "2020-02-12 21:03:47,476 Epoch  63: total training loss 3407.00\n",
      "2020-02-12 21:03:47,476 EPOCH 64\n",
      "2020-02-12 21:03:47,971 Epoch  64 Step:   154900 Batch Loss:     1.261513 Tokens per Sec:     8422, Lr: 0.000300\n",
      "2020-02-12 21:03:58,058 Epoch  64 Step:   155000 Batch Loss:     1.408279 Tokens per Sec:    22280, Lr: 0.000300\n",
      "2020-02-12 21:04:17,159 Example #0\n",
      "2020-02-12 21:04:17,159 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:04:17,159 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:04:17,159 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea lu a dooshima u a lu a mi la her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:04:17,159 Example #1\n",
      "2020-02-12 21:04:17,159 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:04:17,159 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:04:17,159 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:04:17,160 Example #2\n",
      "2020-02-12 21:04:17,160 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:04:17,160 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:04:17,160 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:04:17,160 Example #3\n",
      "2020-02-12 21:04:17,160 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:04:17,160 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:04:17,160 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:04:17,160 Validation result (greedy) at epoch  64, step   155000: bleu:  27.87, loss: 41038.8828, ppl:   3.9999, duration: 19.1018s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:04:27,225 Epoch  64 Step:   155100 Batch Loss:     1.198730 Tokens per Sec:    21476, Lr: 0.000300\n",
      "2020-02-12 21:04:37,239 Epoch  64 Step:   155200 Batch Loss:     1.229787 Tokens per Sec:    22335, Lr: 0.000300\n",
      "2020-02-12 21:04:47,198 Epoch  64 Step:   155300 Batch Loss:     1.316694 Tokens per Sec:    21947, Lr: 0.000300\n",
      "2020-02-12 21:04:57,130 Epoch  64 Step:   155400 Batch Loss:     1.465723 Tokens per Sec:    22158, Lr: 0.000300\n",
      "2020-02-12 21:05:07,238 Epoch  64 Step:   155500 Batch Loss:     1.254294 Tokens per Sec:    21631, Lr: 0.000300\n",
      "2020-02-12 21:05:17,387 Epoch  64 Step:   155600 Batch Loss:     1.200530 Tokens per Sec:    21566, Lr: 0.000300\n",
      "2020-02-12 21:05:27,401 Epoch  64 Step:   155700 Batch Loss:     1.324548 Tokens per Sec:    21785, Lr: 0.000300\n",
      "2020-02-12 21:05:37,339 Epoch  64 Step:   155800 Batch Loss:     1.348824 Tokens per Sec:    22419, Lr: 0.000300\n",
      "2020-02-12 21:05:47,241 Epoch  64 Step:   155900 Batch Loss:     1.429127 Tokens per Sec:    22846, Lr: 0.000300\n",
      "2020-02-12 21:05:57,108 Epoch  64 Step:   156000 Batch Loss:     1.327122 Tokens per Sec:    22629, Lr: 0.000300\n",
      "2020-02-12 21:06:16,339 Example #0\n",
      "2020-02-12 21:06:16,339 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:06:16,339 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:06:16,339 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:06:16,339 Example #1\n",
      "2020-02-12 21:06:16,339 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:06:16,339 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:06:16,339 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man sha kpôô yô , ngu ikyua ! — Rom .\n",
      "2020-02-12 21:06:16,339 Example #2\n",
      "2020-02-12 21:06:16,340 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:06:16,340 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:06:16,340 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:06:16,340 Example #3\n",
      "2020-02-12 21:06:16,340 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:06:16,340 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:06:16,340 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:06:16,340 Validation result (greedy) at epoch  64, step   156000: bleu:  28.26, loss: 40912.4258, ppl:   3.9828, duration: 19.2313s\n",
      "2020-02-12 21:06:26,434 Epoch  64 Step:   156100 Batch Loss:     1.532151 Tokens per Sec:    22016, Lr: 0.000300\n",
      "2020-02-12 21:06:36,523 Epoch  64 Step:   156200 Batch Loss:     1.332851 Tokens per Sec:    22256, Lr: 0.000300\n",
      "2020-02-12 21:06:46,697 Epoch  64 Step:   156300 Batch Loss:     1.391154 Tokens per Sec:    21155, Lr: 0.000300\n",
      "2020-02-12 21:06:56,802 Epoch  64 Step:   156400 Batch Loss:     1.274338 Tokens per Sec:    22371, Lr: 0.000300\n",
      "2020-02-12 21:07:06,936 Epoch  64 Step:   156500 Batch Loss:     1.368083 Tokens per Sec:    21824, Lr: 0.000300\n",
      "2020-02-12 21:07:17,012 Epoch  64 Step:   156600 Batch Loss:     1.492815 Tokens per Sec:    21529, Lr: 0.000300\n",
      "2020-02-12 21:07:27,010 Epoch  64 Step:   156700 Batch Loss:     1.218047 Tokens per Sec:    21941, Lr: 0.000300\n",
      "2020-02-12 21:07:36,953 Epoch  64 Step:   156800 Batch Loss:     1.309190 Tokens per Sec:    21922, Lr: 0.000300\n",
      "2020-02-12 21:07:46,903 Epoch  64 Step:   156900 Batch Loss:     1.275940 Tokens per Sec:    21968, Lr: 0.000300\n",
      "2020-02-12 21:07:57,071 Epoch  64 Step:   157000 Batch Loss:     1.631502 Tokens per Sec:    21923, Lr: 0.000300\n",
      "2020-02-12 21:08:16,694 Example #0\n",
      "2020-02-12 21:08:16,695 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:08:16,695 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:08:16,695 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea lu a dooshima u a lu a mi la her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:08:16,695 Example #1\n",
      "2020-02-12 21:08:16,695 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:08:16,695 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:08:16,695 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:08:16,695 Example #2\n",
      "2020-02-12 21:08:16,695 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:08:16,696 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:08:16,696 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:08:16,696 Example #3\n",
      "2020-02-12 21:08:16,696 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:08:16,696 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:08:16,696 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 21:08:16,696 Validation result (greedy) at epoch  64, step   157000: bleu:  28.19, loss: 40818.1602, ppl:   3.9702, duration: 19.6246s\n",
      "2020-02-12 21:08:26,755 Epoch  64 Step:   157100 Batch Loss:     1.587333 Tokens per Sec:    22398, Lr: 0.000300\n",
      "2020-02-12 21:08:36,816 Epoch  64 Step:   157200 Batch Loss:     1.306947 Tokens per Sec:    22021, Lr: 0.000300\n",
      "2020-02-12 21:08:46,998 Epoch  64 Step:   157300 Batch Loss:     1.558973 Tokens per Sec:    21941, Lr: 0.000300\n",
      "2020-02-12 21:08:52,508 Epoch  64: total training loss 3400.82\n",
      "2020-02-12 21:08:52,509 EPOCH 65\n",
      "2020-02-12 21:08:57,252 Epoch  65 Step:   157400 Batch Loss:     1.531459 Tokens per Sec:    20856, Lr: 0.000300\n",
      "2020-02-12 21:09:07,177 Epoch  65 Step:   157500 Batch Loss:     1.326440 Tokens per Sec:    22359, Lr: 0.000300\n",
      "2020-02-12 21:09:17,110 Epoch  65 Step:   157600 Batch Loss:     1.187809 Tokens per Sec:    22168, Lr: 0.000300\n",
      "2020-02-12 21:09:27,104 Epoch  65 Step:   157700 Batch Loss:     1.356242 Tokens per Sec:    21862, Lr: 0.000300\n",
      "2020-02-12 21:09:37,038 Epoch  65 Step:   157800 Batch Loss:     1.101056 Tokens per Sec:    22262, Lr: 0.000300\n",
      "2020-02-12 21:09:47,041 Epoch  65 Step:   157900 Batch Loss:     1.418730 Tokens per Sec:    22052, Lr: 0.000300\n",
      "2020-02-12 21:09:57,061 Epoch  65 Step:   158000 Batch Loss:     1.318812 Tokens per Sec:    21784, Lr: 0.000300\n",
      "2020-02-12 21:10:15,923 Example #0\n",
      "2020-02-12 21:10:15,923 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:10:15,924 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:10:15,924 \tHypothesis: Er Yehova a ver ivese i lu kwagh u tsan gbem yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u ve lu a mi la , shi vea za hemen u lun a dooshima u a lu vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:10:15,924 Example #1\n",
      "2020-02-12 21:10:15,924 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:10:15,924 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:10:15,924 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:10:15,924 Example #2\n",
      "2020-02-12 21:10:15,924 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:10:15,924 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:10:15,924 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:10:15,924 Example #3\n",
      "2020-02-12 21:10:15,925 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:10:15,925 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:10:15,925 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 21:10:15,925 Validation result (greedy) at epoch  65, step   158000: bleu:  28.04, loss: 40989.2656, ppl:   3.9932, duration: 18.8636s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:10:25,797 Epoch  65 Step:   158100 Batch Loss:     1.485911 Tokens per Sec:    23178, Lr: 0.000300\n",
      "2020-02-12 21:10:35,757 Epoch  65 Step:   158200 Batch Loss:     1.432350 Tokens per Sec:    22102, Lr: 0.000300\n",
      "2020-02-12 21:10:45,728 Epoch  65 Step:   158300 Batch Loss:     1.526295 Tokens per Sec:    22660, Lr: 0.000300\n",
      "2020-02-12 21:10:55,886 Epoch  65 Step:   158400 Batch Loss:     1.449861 Tokens per Sec:    21815, Lr: 0.000300\n",
      "2020-02-12 21:11:06,038 Epoch  65 Step:   158500 Batch Loss:     1.459987 Tokens per Sec:    21674, Lr: 0.000300\n",
      "2020-02-12 21:11:15,960 Epoch  65 Step:   158600 Batch Loss:     1.538805 Tokens per Sec:    22077, Lr: 0.000300\n",
      "2020-02-12 21:11:25,910 Epoch  65 Step:   158700 Batch Loss:     1.522396 Tokens per Sec:    22325, Lr: 0.000300\n",
      "2020-02-12 21:11:35,885 Epoch  65 Step:   158800 Batch Loss:     1.183915 Tokens per Sec:    22091, Lr: 0.000300\n",
      "2020-02-12 21:11:45,882 Epoch  65 Step:   158900 Batch Loss:     1.379419 Tokens per Sec:    21577, Lr: 0.000300\n",
      "2020-02-12 21:11:55,829 Epoch  65 Step:   159000 Batch Loss:     1.420055 Tokens per Sec:    21829, Lr: 0.000300\n",
      "2020-02-12 21:12:14,515 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:12:14,515 Saving new checkpoint.\n",
      "2020-02-12 21:12:14,702 Example #0\n",
      "2020-02-12 21:12:14,703 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:12:14,703 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:12:14,703 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu sha dooshima la shi vea za hemen u lun a dooshima sha dooshima la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:12:14,703 Example #1\n",
      "2020-02-12 21:12:14,703 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:12:14,703 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:12:14,703 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ shighe u doon tsung ne , man kape i lu mgbôghom je la ! — Rom .\n",
      "2020-02-12 21:12:14,703 Example #2\n",
      "2020-02-12 21:12:14,704 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:12:14,704 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:12:14,704 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:12:14,704 Example #3\n",
      "2020-02-12 21:12:14,704 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:12:14,704 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:12:14,704 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 21:12:14,704 Validation result (greedy) at epoch  65, step   159000: bleu:  28.25, loss: 40661.4102, ppl:   3.9492, duration: 18.8745s\n",
      "2020-02-12 21:12:24,664 Epoch  65 Step:   159100 Batch Loss:     1.310821 Tokens per Sec:    21999, Lr: 0.000300\n",
      "2020-02-12 21:12:34,568 Epoch  65 Step:   159200 Batch Loss:     1.425569 Tokens per Sec:    22115, Lr: 0.000300\n",
      "2020-02-12 21:12:44,547 Epoch  65 Step:   159300 Batch Loss:     1.488229 Tokens per Sec:    22357, Lr: 0.000300\n",
      "2020-02-12 21:12:54,600 Epoch  65 Step:   159400 Batch Loss:     1.180263 Tokens per Sec:    21914, Lr: 0.000300\n",
      "2020-02-12 21:13:04,479 Epoch  65 Step:   159500 Batch Loss:     1.516739 Tokens per Sec:    22250, Lr: 0.000300\n",
      "2020-02-12 21:13:14,404 Epoch  65 Step:   159600 Batch Loss:     1.351622 Tokens per Sec:    22450, Lr: 0.000300\n",
      "2020-02-12 21:13:24,322 Epoch  65 Step:   159700 Batch Loss:     1.363642 Tokens per Sec:    22469, Lr: 0.000300\n",
      "2020-02-12 21:13:34,218 Epoch  65 Step:   159800 Batch Loss:     1.448468 Tokens per Sec:    22385, Lr: 0.000300\n",
      "2020-02-12 21:13:35,807 Epoch  65: total training loss 3397.49\n",
      "2020-02-12 21:13:35,808 EPOCH 66\n",
      "2020-02-12 21:13:44,426 Epoch  66 Step:   159900 Batch Loss:     1.479743 Tokens per Sec:    21724, Lr: 0.000300\n",
      "2020-02-12 21:13:54,522 Epoch  66 Step:   160000 Batch Loss:     1.443213 Tokens per Sec:    22317, Lr: 0.000300\n",
      "2020-02-12 21:14:13,762 Example #0\n",
      "2020-02-12 21:14:13,762 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:14:13,762 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:14:13,762 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:14:13,762 Example #1\n",
      "2020-02-12 21:14:13,763 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:14:13,763 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:14:13,763 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:14:13,763 Example #2\n",
      "2020-02-12 21:14:13,763 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:14:13,763 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:14:13,763 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:14:13,763 Example #3\n",
      "2020-02-12 21:14:13,763 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:14:13,763 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:14:13,764 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:14:13,764 Validation result (greedy) at epoch  66, step   160000: bleu:  28.60, loss: 40752.8086, ppl:   3.9614, duration: 19.2415s\n",
      "2020-02-12 21:14:23,952 Epoch  66 Step:   160100 Batch Loss:     1.227053 Tokens per Sec:    21764, Lr: 0.000300\n",
      "2020-02-12 21:14:34,124 Epoch  66 Step:   160200 Batch Loss:     1.307204 Tokens per Sec:    21533, Lr: 0.000300\n",
      "2020-02-12 21:14:44,324 Epoch  66 Step:   160300 Batch Loss:     1.475765 Tokens per Sec:    22031, Lr: 0.000300\n",
      "2020-02-12 21:14:54,344 Epoch  66 Step:   160400 Batch Loss:     1.298973 Tokens per Sec:    22187, Lr: 0.000300\n",
      "2020-02-12 21:15:04,404 Epoch  66 Step:   160500 Batch Loss:     1.446144 Tokens per Sec:    21634, Lr: 0.000300\n",
      "2020-02-12 21:15:14,300 Epoch  66 Step:   160600 Batch Loss:     1.567499 Tokens per Sec:    22486, Lr: 0.000300\n",
      "2020-02-12 21:15:24,298 Epoch  66 Step:   160700 Batch Loss:     1.417793 Tokens per Sec:    22325, Lr: 0.000300\n",
      "2020-02-12 21:15:34,227 Epoch  66 Step:   160800 Batch Loss:     1.486055 Tokens per Sec:    22341, Lr: 0.000300\n",
      "2020-02-12 21:15:44,219 Epoch  66 Step:   160900 Batch Loss:     1.145403 Tokens per Sec:    22093, Lr: 0.000300\n",
      "2020-02-12 21:15:54,338 Epoch  66 Step:   161000 Batch Loss:     1.518003 Tokens per Sec:    22066, Lr: 0.000300\n",
      "2020-02-12 21:16:13,449 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:16:13,450 Saving new checkpoint.\n",
      "2020-02-12 21:16:13,641 Example #0\n",
      "2020-02-12 21:16:13,641 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:16:13,641 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:16:13,641 \tHypothesis: Er i ver ivese i Yehova a ver la i lu kwagh u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo vea lu a dooshima sha dooshima shi vea za hemen u lun a dooshima sha dooshima . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:16:13,641 Example #1\n",
      "2020-02-12 21:16:13,642 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:16:13,642 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:16:13,642 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kape i lu mgbôghom je la ! — Rom .\n",
      "2020-02-12 21:16:13,642 Example #2\n",
      "2020-02-12 21:16:13,642 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:16:13,642 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:16:13,642 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:16:13,642 Example #3\n",
      "2020-02-12 21:16:13,642 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:16:13,642 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:16:13,643 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:16:13,643 Validation result (greedy) at epoch  66, step   161000: bleu:  28.57, loss: 40616.0977, ppl:   3.9432, duration: 19.3035s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:16:23,644 Epoch  66 Step:   161100 Batch Loss:     1.236913 Tokens per Sec:    22313, Lr: 0.000300\n",
      "2020-02-12 21:16:33,592 Epoch  66 Step:   161200 Batch Loss:     1.328223 Tokens per Sec:    22132, Lr: 0.000300\n",
      "2020-02-12 21:16:43,592 Epoch  66 Step:   161300 Batch Loss:     1.279949 Tokens per Sec:    21972, Lr: 0.000300\n",
      "2020-02-12 21:16:53,629 Epoch  66 Step:   161400 Batch Loss:     1.409196 Tokens per Sec:    22547, Lr: 0.000300\n",
      "2020-02-12 21:17:03,665 Epoch  66 Step:   161500 Batch Loss:     1.367484 Tokens per Sec:    21781, Lr: 0.000300\n",
      "2020-02-12 21:17:13,651 Epoch  66 Step:   161600 Batch Loss:     1.233068 Tokens per Sec:    22163, Lr: 0.000300\n",
      "2020-02-12 21:17:23,613 Epoch  66 Step:   161700 Batch Loss:     1.384800 Tokens per Sec:    21532, Lr: 0.000300\n",
      "2020-02-12 21:17:33,728 Epoch  66 Step:   161800 Batch Loss:     1.413949 Tokens per Sec:    21887, Lr: 0.000300\n",
      "2020-02-12 21:17:43,738 Epoch  66 Step:   161900 Batch Loss:     1.314624 Tokens per Sec:    21447, Lr: 0.000300\n",
      "2020-02-12 21:17:53,751 Epoch  66 Step:   162000 Batch Loss:     1.467237 Tokens per Sec:    22285, Lr: 0.000300\n",
      "2020-02-12 21:18:12,524 Example #0\n",
      "2020-02-12 21:18:12,525 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:18:12,525 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:18:12,525 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo tsung vea lu a dooshima ve her shi vea za hemen u lun a dooshima her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:18:12,525 Example #1\n",
      "2020-02-12 21:18:12,525 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:18:12,525 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:18:12,525 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ imba shighe u doon tsung ne , man sha kpôô yô , a mgbôghom ! — Rom .\n",
      "2020-02-12 21:18:12,525 Example #2\n",
      "2020-02-12 21:18:12,525 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:18:12,525 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:18:12,526 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:18:12,526 Example #3\n",
      "2020-02-12 21:18:12,526 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:18:12,526 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:18:12,526 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:18:12,526 Validation result (greedy) at epoch  66, step   162000: bleu:  28.84, loss: 40671.1641, ppl:   3.9505, duration: 18.7739s\n",
      "2020-02-12 21:18:22,717 Epoch  66 Step:   162100 Batch Loss:     1.276497 Tokens per Sec:    21295, Lr: 0.000300\n",
      "2020-02-12 21:18:32,938 Epoch  66 Step:   162200 Batch Loss:     1.624056 Tokens per Sec:    21758, Lr: 0.000300\n",
      "2020-02-12 21:18:40,503 Epoch  66: total training loss 3389.93\n",
      "2020-02-12 21:18:40,503 EPOCH 67\n",
      "2020-02-12 21:18:43,478 Epoch  67 Step:   162300 Batch Loss:     1.098501 Tokens per Sec:    18142, Lr: 0.000300\n",
      "2020-02-12 21:18:53,714 Epoch  67 Step:   162400 Batch Loss:     1.166796 Tokens per Sec:    22235, Lr: 0.000300\n",
      "2020-02-12 21:19:03,827 Epoch  67 Step:   162500 Batch Loss:     1.155450 Tokens per Sec:    20764, Lr: 0.000300\n",
      "2020-02-12 21:19:13,907 Epoch  67 Step:   162600 Batch Loss:     1.451800 Tokens per Sec:    21917, Lr: 0.000300\n",
      "2020-02-12 21:19:23,792 Epoch  67 Step:   162700 Batch Loss:     1.516924 Tokens per Sec:    22630, Lr: 0.000300\n",
      "2020-02-12 21:19:33,834 Epoch  67 Step:   162800 Batch Loss:     0.973673 Tokens per Sec:    21947, Lr: 0.000300\n",
      "2020-02-12 21:19:43,937 Epoch  67 Step:   162900 Batch Loss:     1.156549 Tokens per Sec:    21871, Lr: 0.000300\n",
      "2020-02-12 21:19:54,150 Epoch  67 Step:   163000 Batch Loss:     1.393015 Tokens per Sec:    21804, Lr: 0.000300\n",
      "2020-02-12 21:20:13,318 Example #0\n",
      "2020-02-12 21:20:13,318 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:20:13,318 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:20:13,318 \tHypothesis: Er mserakaa u Yehova a ver la a lu u tsôron yô , doo u noov man kasev vea nôngo kpoghuloo vea kura dooshima ve la , shi vea za hemen u lun a dooshima u a lu a mi la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:20:13,318 Example #1\n",
      "2020-02-12 21:20:13,319 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:20:13,319 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:20:13,319 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:20:13,319 Example #2\n",
      "2020-02-12 21:20:13,319 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:20:13,319 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:20:13,319 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:20:13,319 Example #3\n",
      "2020-02-12 21:20:13,319 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:20:13,319 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:20:13,320 \tHypothesis: Alaghga or u nan lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 21:20:13,320 Validation result (greedy) at epoch  67, step   163000: bleu:  28.42, loss: 40756.3789, ppl:   3.9619, duration: 19.1695s\n",
      "2020-02-12 21:20:23,396 Epoch  67 Step:   163100 Batch Loss:     1.459340 Tokens per Sec:    22207, Lr: 0.000300\n",
      "2020-02-12 21:20:33,411 Epoch  67 Step:   163200 Batch Loss:     1.518558 Tokens per Sec:    22136, Lr: 0.000300\n",
      "2020-02-12 21:20:43,370 Epoch  67 Step:   163300 Batch Loss:     1.452899 Tokens per Sec:    22196, Lr: 0.000300\n",
      "2020-02-12 21:20:53,363 Epoch  67 Step:   163400 Batch Loss:     1.435752 Tokens per Sec:    22106, Lr: 0.000300\n",
      "2020-02-12 21:21:03,267 Epoch  67 Step:   163500 Batch Loss:     1.382020 Tokens per Sec:    22135, Lr: 0.000300\n",
      "2020-02-12 21:21:13,211 Epoch  67 Step:   163600 Batch Loss:     1.319530 Tokens per Sec:    22748, Lr: 0.000300\n",
      "2020-02-12 21:21:23,243 Epoch  67 Step:   163700 Batch Loss:     1.488702 Tokens per Sec:    22444, Lr: 0.000300\n",
      "2020-02-12 21:21:33,103 Epoch  67 Step:   163800 Batch Loss:     1.421444 Tokens per Sec:    22712, Lr: 0.000300\n",
      "2020-02-12 21:21:42,963 Epoch  67 Step:   163900 Batch Loss:     1.558003 Tokens per Sec:    22545, Lr: 0.000300\n",
      "2020-02-12 21:21:52,854 Epoch  67 Step:   164000 Batch Loss:     1.308094 Tokens per Sec:    22296, Lr: 0.000300\n",
      "2020-02-12 21:22:11,259 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:22:11,260 Saving new checkpoint.\n",
      "2020-02-12 21:22:11,449 Example #0\n",
      "2020-02-12 21:22:11,449 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:22:11,449 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:22:11,449 \tHypothesis: Er mserakaa u Yehova a ver la a lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea za hemen u lun a dooshima u mimi shi vea za hemen u lun a dooshima sha dooshima . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:22:11,449 Example #1\n",
      "2020-02-12 21:22:11,449 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:22:11,449 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:22:11,450 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:22:11,450 Example #2\n",
      "2020-02-12 21:22:11,450 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:22:11,450 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:22:11,450 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:22:11,450 Example #3\n",
      "2020-02-12 21:22:11,450 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:22:11,450 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:22:11,450 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga yô .\n",
      "2020-02-12 21:22:11,450 Validation result (greedy) at epoch  67, step   164000: bleu:  28.61, loss: 40605.3984, ppl:   3.9417, duration: 18.5963s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:22:21,448 Epoch  67 Step:   164100 Batch Loss:     1.479446 Tokens per Sec:    22097, Lr: 0.000300\n",
      "2020-02-12 21:22:31,338 Epoch  67 Step:   164200 Batch Loss:     1.453567 Tokens per Sec:    22385, Lr: 0.000300\n",
      "2020-02-12 21:22:41,451 Epoch  67 Step:   164300 Batch Loss:     1.358651 Tokens per Sec:    21795, Lr: 0.000300\n",
      "2020-02-12 21:22:51,718 Epoch  67 Step:   164400 Batch Loss:     1.157368 Tokens per Sec:    21548, Lr: 0.000300\n",
      "2020-02-12 21:23:01,888 Epoch  67 Step:   164500 Batch Loss:     1.052091 Tokens per Sec:    21355, Lr: 0.000300\n",
      "2020-02-12 21:23:11,869 Epoch  67 Step:   164600 Batch Loss:     1.465715 Tokens per Sec:    22545, Lr: 0.000300\n",
      "2020-02-12 21:23:21,879 Epoch  67 Step:   164700 Batch Loss:     1.388363 Tokens per Sec:    21592, Lr: 0.000300\n",
      "2020-02-12 21:23:24,985 Epoch  67: total training loss 3381.95\n",
      "2020-02-12 21:23:24,985 EPOCH 68\n",
      "2020-02-12 21:23:32,104 Epoch  68 Step:   164800 Batch Loss:     1.100460 Tokens per Sec:    21014, Lr: 0.000300\n",
      "2020-02-12 21:23:42,042 Epoch  68 Step:   164900 Batch Loss:     1.189202 Tokens per Sec:    22189, Lr: 0.000300\n",
      "2020-02-12 21:23:51,988 Epoch  68 Step:   165000 Batch Loss:     1.570109 Tokens per Sec:    22576, Lr: 0.000300\n",
      "2020-02-12 21:24:10,967 Example #0\n",
      "2020-02-12 21:24:10,968 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:24:10,968 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:24:10,968 \tHypothesis: Er mserakaa u ivesegh u Yehova a ver la a lu u tsan gbem yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima ve her shi vea za hemen u lun a dooshima u a lu seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:24:10,968 Example #1\n",
      "2020-02-12 21:24:10,968 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:24:10,968 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:24:10,968 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man a mgbôghom ! — Rom .\n",
      "2020-02-12 21:24:10,968 Example #2\n",
      "2020-02-12 21:24:10,968 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:24:10,968 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:24:10,968 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:24:10,968 Example #3\n",
      "2020-02-12 21:24:10,969 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:24:10,969 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:24:10,969 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:24:10,969 Validation result (greedy) at epoch  68, step   165000: bleu:  28.36, loss: 40696.5312, ppl:   3.9539, duration: 18.9807s\n",
      "2020-02-12 21:24:21,042 Epoch  68 Step:   165100 Batch Loss:     1.531411 Tokens per Sec:    21828, Lr: 0.000300\n",
      "2020-02-12 21:24:31,002 Epoch  68 Step:   165200 Batch Loss:     1.362218 Tokens per Sec:    22641, Lr: 0.000300\n",
      "2020-02-12 21:24:40,904 Epoch  68 Step:   165300 Batch Loss:     1.406649 Tokens per Sec:    22367, Lr: 0.000300\n",
      "2020-02-12 21:24:50,822 Epoch  68 Step:   165400 Batch Loss:     1.510821 Tokens per Sec:    22328, Lr: 0.000300\n",
      "2020-02-12 21:25:00,698 Epoch  68 Step:   165500 Batch Loss:     1.466789 Tokens per Sec:    21804, Lr: 0.000300\n",
      "2020-02-12 21:25:10,606 Epoch  68 Step:   165600 Batch Loss:     1.211068 Tokens per Sec:    21851, Lr: 0.000300\n",
      "2020-02-12 21:25:20,640 Epoch  68 Step:   165700 Batch Loss:     1.582863 Tokens per Sec:    21871, Lr: 0.000300\n",
      "2020-02-12 21:25:30,889 Epoch  68 Step:   165800 Batch Loss:     1.334930 Tokens per Sec:    21670, Lr: 0.000300\n",
      "2020-02-12 21:25:40,892 Epoch  68 Step:   165900 Batch Loss:     1.591504 Tokens per Sec:    22002, Lr: 0.000300\n",
      "2020-02-12 21:25:50,895 Epoch  68 Step:   166000 Batch Loss:     1.348272 Tokens per Sec:    22290, Lr: 0.000300\n",
      "2020-02-12 21:26:09,744 Example #0\n",
      "2020-02-12 21:26:09,744 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:26:09,744 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:26:09,744 \tHypothesis: Er mserakaa u Yehova a ver la a lu u tsan gbem yô , gba hange hange u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea lu a dooshima u a lu a mi la kpaa . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:26:09,744 Example #1\n",
      "2020-02-12 21:26:09,744 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:26:09,744 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:26:09,744 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man sha kpôô yô , ngu ikyua ! — Rom .\n",
      "2020-02-12 21:26:09,744 Example #2\n",
      "2020-02-12 21:26:09,745 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:26:09,745 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:26:09,745 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:26:09,745 Example #3\n",
      "2020-02-12 21:26:09,745 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:26:09,745 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:26:09,745 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:26:09,745 Validation result (greedy) at epoch  68, step   166000: bleu:  28.40, loss: 40709.0000, ppl:   3.9555, duration: 18.8493s\n",
      "2020-02-12 21:26:19,657 Epoch  68 Step:   166100 Batch Loss:     1.542606 Tokens per Sec:    22114, Lr: 0.000300\n",
      "2020-02-12 21:26:29,520 Epoch  68 Step:   166200 Batch Loss:     1.171417 Tokens per Sec:    22602, Lr: 0.000300\n",
      "2020-02-12 21:26:39,423 Epoch  68 Step:   166300 Batch Loss:     1.482436 Tokens per Sec:    22155, Lr: 0.000300\n",
      "2020-02-12 21:26:49,407 Epoch  68 Step:   166400 Batch Loss:     1.185074 Tokens per Sec:    21844, Lr: 0.000300\n",
      "2020-02-12 21:26:59,370 Epoch  68 Step:   166500 Batch Loss:     1.324131 Tokens per Sec:    21946, Lr: 0.000300\n",
      "2020-02-12 21:27:09,478 Epoch  68 Step:   166600 Batch Loss:     1.267979 Tokens per Sec:    22332, Lr: 0.000300\n",
      "2020-02-12 21:27:19,561 Epoch  68 Step:   166700 Batch Loss:     1.482439 Tokens per Sec:    21716, Lr: 0.000300\n",
      "2020-02-12 21:27:29,771 Epoch  68 Step:   166800 Batch Loss:     1.574959 Tokens per Sec:    21460, Lr: 0.000300\n",
      "2020-02-12 21:27:39,901 Epoch  68 Step:   166900 Batch Loss:     1.184707 Tokens per Sec:    21924, Lr: 0.000300\n",
      "2020-02-12 21:27:50,070 Epoch  68 Step:   167000 Batch Loss:     1.327590 Tokens per Sec:    22552, Lr: 0.000300\n",
      "2020-02-12 21:28:09,927 Example #0\n",
      "2020-02-12 21:28:09,928 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:28:09,928 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:28:09,928 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima her dông . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:28:09,928 Example #1\n",
      "2020-02-12 21:28:09,928 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:28:09,928 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:28:09,928 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u kpilighyol ne , man kwagh ne ngu ikyua ikyua ! — Rom .\n",
      "2020-02-12 21:28:09,928 Example #2\n",
      "2020-02-12 21:28:09,928 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:28:09,929 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:28:09,929 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:28:09,929 Example #3\n",
      "2020-02-12 21:28:09,929 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:28:09,929 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:28:09,929 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:28:09,929 Validation result (greedy) at epoch  68, step   167000: bleu:  28.46, loss: 40678.6406, ppl:   3.9515, duration: 19.8590s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:28:19,756 Epoch  68 Step:   167100 Batch Loss:     1.320471 Tokens per Sec:    22520, Lr: 0.000300\n",
      "2020-02-12 21:28:28,596 Epoch  68: total training loss 3378.79\n",
      "2020-02-12 21:28:28,597 EPOCH 69\n",
      "2020-02-12 21:28:29,860 Epoch  69 Step:   167200 Batch Loss:     1.252517 Tokens per Sec:    19149, Lr: 0.000300\n",
      "2020-02-12 21:28:39,683 Epoch  69 Step:   167300 Batch Loss:     0.909732 Tokens per Sec:    22025, Lr: 0.000300\n",
      "2020-02-12 21:28:49,603 Epoch  69 Step:   167400 Batch Loss:     1.272224 Tokens per Sec:    21858, Lr: 0.000300\n",
      "2020-02-12 21:28:59,517 Epoch  69 Step:   167500 Batch Loss:     1.142943 Tokens per Sec:    21971, Lr: 0.000300\n",
      "2020-02-12 21:29:09,505 Epoch  69 Step:   167600 Batch Loss:     1.449285 Tokens per Sec:    22401, Lr: 0.000300\n",
      "2020-02-12 21:29:19,519 Epoch  69 Step:   167700 Batch Loss:     1.524561 Tokens per Sec:    21455, Lr: 0.000300\n",
      "2020-02-12 21:29:29,495 Epoch  69 Step:   167800 Batch Loss:     1.219512 Tokens per Sec:    22015, Lr: 0.000300\n",
      "2020-02-12 21:29:39,606 Epoch  69 Step:   167900 Batch Loss:     1.298621 Tokens per Sec:    21992, Lr: 0.000300\n",
      "2020-02-12 21:29:49,831 Epoch  69 Step:   168000 Batch Loss:     1.412660 Tokens per Sec:    21519, Lr: 0.000300\n",
      "2020-02-12 21:30:09,305 Example #0\n",
      "2020-02-12 21:30:09,305 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:30:09,305 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:30:09,305 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsôron yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima her . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:30:09,305 Example #1\n",
      "2020-02-12 21:30:09,305 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:30:09,305 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:30:09,306 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:30:09,306 Example #2\n",
      "2020-02-12 21:30:09,306 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:30:09,306 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:30:09,306 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:30:09,306 Example #3\n",
      "2020-02-12 21:30:09,306 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:30:09,306 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:30:09,306 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:30:09,306 Validation result (greedy) at epoch  69, step   168000: bleu:  28.71, loss: 40764.4219, ppl:   3.9630, duration: 19.4748s\n",
      "2020-02-12 21:30:19,537 Epoch  69 Step:   168100 Batch Loss:     1.460573 Tokens per Sec:    21243, Lr: 0.000300\n",
      "2020-02-12 21:30:29,645 Epoch  69 Step:   168200 Batch Loss:     1.524707 Tokens per Sec:    21518, Lr: 0.000300\n",
      "2020-02-12 21:30:39,756 Epoch  69 Step:   168300 Batch Loss:     1.495079 Tokens per Sec:    21501, Lr: 0.000300\n",
      "2020-02-12 21:30:49,913 Epoch  69 Step:   168400 Batch Loss:     1.492542 Tokens per Sec:    22284, Lr: 0.000300\n",
      "2020-02-12 21:30:59,872 Epoch  69 Step:   168500 Batch Loss:     1.407108 Tokens per Sec:    22497, Lr: 0.000300\n",
      "2020-02-12 21:31:09,729 Epoch  69 Step:   168600 Batch Loss:     1.392927 Tokens per Sec:    22555, Lr: 0.000300\n",
      "2020-02-12 21:31:19,609 Epoch  69 Step:   168700 Batch Loss:     1.501685 Tokens per Sec:    22699, Lr: 0.000300\n",
      "2020-02-12 21:31:29,534 Epoch  69 Step:   168800 Batch Loss:     1.412920 Tokens per Sec:    22659, Lr: 0.000300\n",
      "2020-02-12 21:31:39,495 Epoch  69 Step:   168900 Batch Loss:     0.937086 Tokens per Sec:    22502, Lr: 0.000300\n",
      "2020-02-12 21:31:49,543 Epoch  69 Step:   169000 Batch Loss:     1.586249 Tokens per Sec:    21540, Lr: 0.000300\n",
      "2020-02-12 21:32:08,817 Example #0\n",
      "2020-02-12 21:32:08,817 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:32:08,818 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:32:08,818 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem sha won yô , doo u noov man kasev vea nôngo sha afatyô ve cii vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:32:08,818 Example #1\n",
      "2020-02-12 21:32:08,818 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:32:08,818 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:32:08,818 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne ngu ikyua ! — Rom .\n",
      "2020-02-12 21:32:08,818 Example #2\n",
      "2020-02-12 21:32:08,818 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:32:08,818 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:32:08,818 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu ngu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:32:08,818 Example #3\n",
      "2020-02-12 21:32:08,819 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:32:08,819 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:32:08,819 \tHypothesis: Alaghga or u nan lu eren kwagh sha inja ga yô , nana lu eren kwagh u vough .\n",
      "2020-02-12 21:32:08,819 Validation result (greedy) at epoch  69, step   169000: bleu:  29.16, loss: 40690.5859, ppl:   3.9531, duration: 19.2749s\n",
      "2020-02-12 21:32:18,751 Epoch  69 Step:   169100 Batch Loss:     1.425263 Tokens per Sec:    22242, Lr: 0.000300\n",
      "2020-02-12 21:32:28,633 Epoch  69 Step:   169200 Batch Loss:     1.347558 Tokens per Sec:    22792, Lr: 0.000300\n",
      "2020-02-12 21:32:38,602 Epoch  69 Step:   169300 Batch Loss:     1.265322 Tokens per Sec:    22441, Lr: 0.000300\n",
      "2020-02-12 21:32:48,753 Epoch  69 Step:   169400 Batch Loss:     1.470473 Tokens per Sec:    21936, Lr: 0.000300\n",
      "2020-02-12 21:32:58,666 Epoch  69 Step:   169500 Batch Loss:     1.598056 Tokens per Sec:    22383, Lr: 0.000300\n",
      "2020-02-12 21:33:08,609 Epoch  69 Step:   169600 Batch Loss:     1.198459 Tokens per Sec:    22573, Lr: 0.000300\n",
      "2020-02-12 21:33:13,319 Epoch  69: total training loss 3371.32\n",
      "2020-02-12 21:33:13,319 EPOCH 70\n",
      "2020-02-12 21:33:18,848 Epoch  70 Step:   169700 Batch Loss:     1.480597 Tokens per Sec:    20277, Lr: 0.000300\n",
      "2020-02-12 21:33:28,848 Epoch  70 Step:   169800 Batch Loss:     1.431355 Tokens per Sec:    22720, Lr: 0.000300\n",
      "2020-02-12 21:33:38,740 Epoch  70 Step:   169900 Batch Loss:     1.516980 Tokens per Sec:    22351, Lr: 0.000300\n",
      "2020-02-12 21:33:48,774 Epoch  70 Step:   170000 Batch Loss:     1.378151 Tokens per Sec:    22478, Lr: 0.000300\n",
      "2020-02-12 21:34:06,979 Example #0\n",
      "2020-02-12 21:34:06,980 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:34:06,980 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:34:06,980 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve la , shi vea za hemen u lun a dooshima u a lu vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:34:06,980 Example #1\n",
      "2020-02-12 21:34:06,980 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:34:06,980 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:34:06,980 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:34:06,980 Example #2\n",
      "2020-02-12 21:34:06,980 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:34:06,980 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:34:06,980 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:34:06,981 Example #3\n",
      "2020-02-12 21:34:06,981 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:34:06,981 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:34:06,981 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:34:06,981 Validation result (greedy) at epoch  70, step   170000: bleu:  28.13, loss: 40766.2070, ppl:   3.9632, duration: 18.2060s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:34:16,872 Epoch  70 Step:   170100 Batch Loss:     1.500207 Tokens per Sec:    21710, Lr: 0.000210\n",
      "2020-02-12 21:34:26,790 Epoch  70 Step:   170200 Batch Loss:     1.456264 Tokens per Sec:    22478, Lr: 0.000210\n",
      "2020-02-12 21:34:36,775 Epoch  70 Step:   170300 Batch Loss:     1.225434 Tokens per Sec:    22160, Lr: 0.000210\n",
      "2020-02-12 21:34:46,674 Epoch  70 Step:   170400 Batch Loss:     1.147439 Tokens per Sec:    22633, Lr: 0.000210\n",
      "2020-02-12 21:34:56,574 Epoch  70 Step:   170500 Batch Loss:     1.492128 Tokens per Sec:    21642, Lr: 0.000210\n",
      "2020-02-12 21:35:06,698 Epoch  70 Step:   170600 Batch Loss:     1.142891 Tokens per Sec:    21335, Lr: 0.000210\n",
      "2020-02-12 21:35:16,806 Epoch  70 Step:   170700 Batch Loss:     1.164842 Tokens per Sec:    21830, Lr: 0.000210\n",
      "2020-02-12 21:35:26,878 Epoch  70 Step:   170800 Batch Loss:     1.104715 Tokens per Sec:    22225, Lr: 0.000210\n",
      "2020-02-12 21:35:37,001 Epoch  70 Step:   170900 Batch Loss:     1.374452 Tokens per Sec:    21498, Lr: 0.000210\n",
      "2020-02-12 21:35:46,975 Epoch  70 Step:   171000 Batch Loss:     1.248356 Tokens per Sec:    22564, Lr: 0.000210\n",
      "2020-02-12 21:36:05,576 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:36:05,576 Saving new checkpoint.\n",
      "2020-02-12 21:36:05,765 Example #0\n",
      "2020-02-12 21:36:05,765 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:36:05,765 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:36:05,765 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve her shi vea za hemen u lun a dooshima u a lu seer a seer la . — Mar . 10 : 6 - 9 .\n",
      "2020-02-12 21:36:05,765 Example #1\n",
      "2020-02-12 21:36:05,765 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:36:05,765 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:36:05,766 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:36:05,766 Example #2\n",
      "2020-02-12 21:36:05,766 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:36:05,766 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:36:05,766 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:36:05,766 Example #3\n",
      "2020-02-12 21:36:05,766 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:36:05,766 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:36:05,766 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:36:05,766 Validation result (greedy) at epoch  70, step   171000: bleu:  28.70, loss: 40408.3711, ppl:   3.9156, duration: 18.7912s\n",
      "2020-02-12 21:36:15,736 Epoch  70 Step:   171100 Batch Loss:     1.453360 Tokens per Sec:    22331, Lr: 0.000210\n",
      "2020-02-12 21:36:25,862 Epoch  70 Step:   171200 Batch Loss:     1.255177 Tokens per Sec:    21728, Lr: 0.000210\n",
      "2020-02-12 21:36:36,096 Epoch  70 Step:   171300 Batch Loss:     1.478583 Tokens per Sec:    21594, Lr: 0.000210\n",
      "2020-02-12 21:36:46,228 Epoch  70 Step:   171400 Batch Loss:     1.453074 Tokens per Sec:    21447, Lr: 0.000210\n",
      "2020-02-12 21:36:56,243 Epoch  70 Step:   171500 Batch Loss:     1.361747 Tokens per Sec:    22216, Lr: 0.000210\n",
      "2020-02-12 21:37:06,194 Epoch  70 Step:   171600 Batch Loss:     1.390606 Tokens per Sec:    22607, Lr: 0.000210\n",
      "2020-02-12 21:37:16,141 Epoch  70 Step:   171700 Batch Loss:     1.289407 Tokens per Sec:    22233, Lr: 0.000210\n",
      "2020-02-12 21:37:26,056 Epoch  70 Step:   171800 Batch Loss:     1.517298 Tokens per Sec:    22360, Lr: 0.000210\n",
      "2020-02-12 21:37:35,929 Epoch  70 Step:   171900 Batch Loss:     1.117063 Tokens per Sec:    22192, Lr: 0.000210\n",
      "2020-02-12 21:37:45,910 Epoch  70 Step:   172000 Batch Loss:     1.226853 Tokens per Sec:    22136, Lr: 0.000210\n",
      "2020-02-12 21:38:04,465 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:38:04,465 Saving new checkpoint.\n",
      "2020-02-12 21:38:04,652 Example #0\n",
      "2020-02-12 21:38:04,653 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:38:04,653 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:38:04,653 \tHypothesis: Er mserakaa u Yehova a ver la a lu u tsan gbem sha won yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu vesen seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:38:04,653 Example #1\n",
      "2020-02-12 21:38:04,653 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:38:04,653 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:38:04,653 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:38:04,653 Example #2\n",
      "2020-02-12 21:38:04,653 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:38:04,653 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:38:04,653 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:38:04,654 Example #3\n",
      "2020-02-12 21:38:04,654 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:38:04,654 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:38:04,654 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:38:04,654 Validation result (greedy) at epoch  70, step   172000: bleu:  28.21, loss: 40373.5430, ppl:   3.9110, duration: 18.7433s\n",
      "2020-02-12 21:38:14,603 Epoch  70 Step:   172100 Batch Loss:     1.398068 Tokens per Sec:    21901, Lr: 0.000210\n",
      "2020-02-12 21:38:15,500 Epoch  70: total training loss 3335.88\n",
      "2020-02-12 21:38:15,501 EPOCH 71\n",
      "2020-02-12 21:38:24,834 Epoch  71 Step:   172200 Batch Loss:     1.417088 Tokens per Sec:    21670, Lr: 0.000210\n",
      "2020-02-12 21:38:34,720 Epoch  71 Step:   172300 Batch Loss:     1.470526 Tokens per Sec:    22203, Lr: 0.000210\n",
      "2020-02-12 21:38:44,617 Epoch  71 Step:   172400 Batch Loss:     1.138673 Tokens per Sec:    22724, Lr: 0.000210\n",
      "2020-02-12 21:38:54,544 Epoch  71 Step:   172500 Batch Loss:     1.186428 Tokens per Sec:    22289, Lr: 0.000210\n",
      "2020-02-12 21:39:04,443 Epoch  71 Step:   172600 Batch Loss:     1.258686 Tokens per Sec:    22818, Lr: 0.000210\n",
      "2020-02-12 21:39:14,387 Epoch  71 Step:   172700 Batch Loss:     1.423118 Tokens per Sec:    22370, Lr: 0.000210\n",
      "2020-02-12 21:39:24,414 Epoch  71 Step:   172800 Batch Loss:     1.401284 Tokens per Sec:    22209, Lr: 0.000210\n",
      "2020-02-12 21:39:34,317 Epoch  71 Step:   172900 Batch Loss:     1.374153 Tokens per Sec:    22307, Lr: 0.000210\n",
      "2020-02-12 21:39:44,285 Epoch  71 Step:   173000 Batch Loss:     1.007435 Tokens per Sec:    22367, Lr: 0.000210\n",
      "2020-02-12 21:40:02,477 Example #0\n",
      "2020-02-12 21:40:02,477 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:40:02,478 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:40:02,478 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem yô , doo u noov man kasev vea nôngo kpoghuloo vea lu a dooshima ve her shi vea za hemen u lun a dooshima u a lu vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:40:02,478 Example #1\n",
      "2020-02-12 21:40:02,478 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:40:02,478 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:40:02,478 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man ka shighe u a mgbôghom je la ! — Rom .\n",
      "2020-02-12 21:40:02,478 Example #2\n",
      "2020-02-12 21:40:02,478 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:40:02,478 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:40:02,478 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:40:02,478 Example #3\n",
      "2020-02-12 21:40:02,479 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:40:02,479 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:40:02,479 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:40:02,479 Validation result (greedy) at epoch  71, step   173000: bleu:  28.49, loss: 40381.0352, ppl:   3.9120, duration: 18.1928s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:40:12,345 Epoch  71 Step:   173100 Batch Loss:     1.539012 Tokens per Sec:    21999, Lr: 0.000210\n",
      "2020-02-12 21:40:22,200 Epoch  71 Step:   173200 Batch Loss:     1.469249 Tokens per Sec:    22888, Lr: 0.000210\n",
      "2020-02-12 21:40:32,063 Epoch  71 Step:   173300 Batch Loss:     1.316316 Tokens per Sec:    22537, Lr: 0.000210\n",
      "2020-02-12 21:40:41,952 Epoch  71 Step:   173400 Batch Loss:     1.058633 Tokens per Sec:    21707, Lr: 0.000210\n",
      "2020-02-12 21:40:51,943 Epoch  71 Step:   173500 Batch Loss:     1.452735 Tokens per Sec:    22243, Lr: 0.000210\n",
      "2020-02-12 21:41:01,866 Epoch  71 Step:   173600 Batch Loss:     1.478271 Tokens per Sec:    22119, Lr: 0.000210\n",
      "2020-02-12 21:41:11,714 Epoch  71 Step:   173700 Batch Loss:     1.305297 Tokens per Sec:    21938, Lr: 0.000210\n",
      "2020-02-12 21:41:21,573 Epoch  71 Step:   173800 Batch Loss:     1.453498 Tokens per Sec:    22531, Lr: 0.000210\n",
      "2020-02-12 21:41:31,473 Epoch  71 Step:   173900 Batch Loss:     1.310819 Tokens per Sec:    22287, Lr: 0.000210\n",
      "2020-02-12 21:41:41,381 Epoch  71 Step:   174000 Batch Loss:     1.476677 Tokens per Sec:    22879, Lr: 0.000210\n",
      "2020-02-12 21:42:00,421 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:42:00,421 Saving new checkpoint.\n",
      "2020-02-12 21:42:00,620 Example #0\n",
      "2020-02-12 21:42:00,620 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:42:00,620 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:42:00,620 \tHypothesis: Er mserakaa u Yehova a ver la a lu u tsan gbem sha won yô , doo u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu vesen seer a seer la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:42:00,620 Example #1\n",
      "2020-02-12 21:42:00,620 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:42:00,620 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:42:00,621 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:42:00,621 Example #2\n",
      "2020-02-12 21:42:00,621 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:42:00,621 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:42:00,621 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:42:00,621 Example #3\n",
      "2020-02-12 21:42:00,621 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:42:00,621 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:42:00,621 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:42:00,621 Validation result (greedy) at epoch  71, step   174000: bleu:  28.97, loss: 40309.5039, ppl:   3.9025, duration: 19.2404s\n",
      "2020-02-12 21:42:10,694 Epoch  71 Step:   174100 Batch Loss:     1.381948 Tokens per Sec:    22018, Lr: 0.000210\n",
      "2020-02-12 21:42:20,684 Epoch  71 Step:   174200 Batch Loss:     1.502970 Tokens per Sec:    21835, Lr: 0.000210\n",
      "2020-02-12 21:42:30,802 Epoch  71 Step:   174300 Batch Loss:     1.162560 Tokens per Sec:    22363, Lr: 0.000210\n",
      "2020-02-12 21:42:40,801 Epoch  71 Step:   174400 Batch Loss:     1.131786 Tokens per Sec:    22201, Lr: 0.000210\n",
      "2020-02-12 21:42:50,770 Epoch  71 Step:   174500 Batch Loss:     1.192420 Tokens per Sec:    22214, Lr: 0.000210\n",
      "2020-02-12 21:42:56,954 Epoch  71: total training loss 3297.80\n",
      "2020-02-12 21:42:56,954 EPOCH 72\n",
      "2020-02-12 21:43:01,049 Epoch  72 Step:   174600 Batch Loss:     1.326846 Tokens per Sec:    21358, Lr: 0.000210\n",
      "2020-02-12 21:43:11,000 Epoch  72 Step:   174700 Batch Loss:     1.642390 Tokens per Sec:    22619, Lr: 0.000210\n",
      "2020-02-12 21:43:21,045 Epoch  72 Step:   174800 Batch Loss:     1.298363 Tokens per Sec:    21769, Lr: 0.000210\n",
      "2020-02-12 21:43:31,130 Epoch  72 Step:   174900 Batch Loss:     1.373066 Tokens per Sec:    21830, Lr: 0.000210\n",
      "2020-02-12 21:43:41,315 Epoch  72 Step:   175000 Batch Loss:     1.351639 Tokens per Sec:    22527, Lr: 0.000210\n",
      "2020-02-12 21:44:00,909 Example #0\n",
      "2020-02-12 21:44:00,909 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:44:00,909 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:44:00,909 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem sha won yô , gba hange hange u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u una fatyô u vesen la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:44:00,909 Example #1\n",
      "2020-02-12 21:44:00,910 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:44:00,910 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:44:00,910 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne , man sha kpôô yô , ngu ikyua ! — Rom .\n",
      "2020-02-12 21:44:00,910 Example #2\n",
      "2020-02-12 21:44:00,910 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:44:00,910 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:44:00,910 \tHypothesis: Anyom nga kar pue yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:44:00,910 Example #3\n",
      "2020-02-12 21:44:00,910 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:44:00,910 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:44:00,910 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:44:00,910 Validation result (greedy) at epoch  72, step   175000: bleu:  28.55, loss: 40399.0703, ppl:   3.9143, duration: 19.5949s\n",
      "2020-02-12 21:44:11,070 Epoch  72 Step:   175100 Batch Loss:     1.495386 Tokens per Sec:    21784, Lr: 0.000210\n",
      "2020-02-12 21:44:21,192 Epoch  72 Step:   175200 Batch Loss:     1.402765 Tokens per Sec:    21802, Lr: 0.000210\n",
      "2020-02-12 21:44:31,226 Epoch  72 Step:   175300 Batch Loss:     1.395334 Tokens per Sec:    22162, Lr: 0.000210\n",
      "2020-02-12 21:44:41,101 Epoch  72 Step:   175400 Batch Loss:     1.064630 Tokens per Sec:    22288, Lr: 0.000210\n",
      "2020-02-12 21:44:51,030 Epoch  72 Step:   175500 Batch Loss:     1.288134 Tokens per Sec:    22433, Lr: 0.000210\n",
      "2020-02-12 21:45:00,873 Epoch  72 Step:   175600 Batch Loss:     0.982788 Tokens per Sec:    22172, Lr: 0.000210\n",
      "2020-02-12 21:45:10,765 Epoch  72 Step:   175700 Batch Loss:     1.539751 Tokens per Sec:    22442, Lr: 0.000210\n",
      "2020-02-12 21:45:20,755 Epoch  72 Step:   175800 Batch Loss:     1.439170 Tokens per Sec:    22132, Lr: 0.000210\n",
      "2020-02-12 21:45:30,833 Epoch  72 Step:   175900 Batch Loss:     1.477820 Tokens per Sec:    21847, Lr: 0.000210\n",
      "2020-02-12 21:45:40,920 Epoch  72 Step:   176000 Batch Loss:     1.483829 Tokens per Sec:    21566, Lr: 0.000210\n",
      "2020-02-12 21:46:00,476 Hooray! New best validation result [ppl]!\n",
      "2020-02-12 21:46:00,476 Saving new checkpoint.\n",
      "2020-02-12 21:46:00,667 Example #0\n",
      "2020-02-12 21:46:00,668 \tSource:     Since the marriage arrangement instituted by Jehovah is a lasting one , it is vital that couples endeavor to keep the flame of their love ablaze and maintain an atmosphere in which love can grow . ​ — Mark 10 : 6 - 9 .\n",
      "2020-02-12 21:46:00,668 \tReference:  10 : ​ 6 - 9 .\n",
      "2020-02-12 21:46:00,668 \tHypothesis: Er ivese i Yehova a ver la i lu kwagh u tsan gbem yô , doo u noov man kasev vea nôngo kpoghuloo vea lu a dooshima u a lu a mi la , shi vea za hemen u lun a dooshima u a lu a mi la . — Marku 10 : 6 - 9 .\n",
      "2020-02-12 21:46:00,668 Example #1\n",
      "2020-02-12 21:46:00,668 \tSource:     We too can “ see ” such a wonderful time ​ — and it is near indeed ! ​ — Rom .\n",
      "2020-02-12 21:46:00,668 \tReference:  Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung ne ken ishima yase , shi ngu ica a vese kpaa ga ! ​ — Rom .\n",
      "2020-02-12 21:46:00,668 \tHypothesis: Se kpa se fatyô u ‘ nengen ’ a shighe u doon tsung , man kwagh ne mgbôghom ! — Rom .\n",
      "2020-02-12 21:46:00,668 Example #2\n",
      "2020-02-12 21:46:00,668 \tSource:     About ten years later , Paul is in Macedonia and Timothy is in Ephesus .\n",
      "2020-02-12 21:46:00,668 \tReference:  Anyom pue nga kar yô , Paulu yem ken Masedonia , Timoteu di lu ken Efese .\n",
      "2020-02-12 21:46:00,669 \tHypothesis: Anyom nga kar er pue nahan yô , Paulu lu ken Masedonia man Timoteu ken Efese .\n",
      "2020-02-12 21:46:00,669 Example #3\n",
      "2020-02-12 21:46:00,669 \tSource:     It could be that an individual is acting in a way that is not appropriate .\n",
      "2020-02-12 21:46:00,669 \tReference:  Alaghga ma anmgbian ngu eren inja ka doon ga .\n",
      "2020-02-12 21:46:00,669 \tHypothesis: Alaghga or nana lu eren kwagh sha gbenda u vough ga .\n",
      "2020-02-12 21:46:00,669 Validation result (greedy) at epoch  72, step   176000: bleu:  28.60, loss: 40274.0742, ppl:   3.8979, duration: 19.7486s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-12 21:46:10,788 Epoch  72 Step:   176100 Batch Loss:     1.370144 Tokens per Sec:    21528, Lr: 0.000210\n",
      "2020-02-12 21:46:20,968 Epoch  72 Step:   176200 Batch Loss:     1.523032 Tokens per Sec:    21737, Lr: 0.000210\n",
      "2020-02-12 21:46:31,047 Epoch  72 Step:   176300 Batch Loss:     1.417220 Tokens per Sec:    22094, Lr: 0.000210\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p \"$experiment_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$experiment_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n94wlrCjVc17"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Steps: 1000\tLoss: 105431.41406\tPPL: 35.21214\tbleu: 2.23576\tLR: 0.00030000\t*\r\n",
      "Steps: 2000\tLoss: 90648.33594\tPPL: 21.37090\tbleu: 3.87765\tLR: 0.00030000\t*\r\n",
      "Steps: 3000\tLoss: 83058.23438\tPPL: 16.53769\tbleu: 4.98396\tLR: 0.00030000\t*\r\n",
      "Steps: 4000\tLoss: 77589.78125\tPPL: 13.74840\tbleu: 7.26332\tLR: 0.00030000\t*\r\n",
      "Steps: 5000\tLoss: 73523.78906\tPPL: 11.98404\tbleu: 9.31015\tLR: 0.00030000\t*\r\n",
      "Steps: 6000\tLoss: 70898.39062\tPPL: 10.96702\tbleu: 10.50791\tLR: 0.00030000\t*\r\n",
      "Steps: 7000\tLoss: 68224.22656\tPPL: 10.01978\tbleu: 11.86918\tLR: 0.00030000\t*\r\n",
      "Steps: 8000\tLoss: 66128.57031\tPPL: 9.33501\tbleu: 13.12602\tLR: 0.00030000\t*\r\n",
      "Steps: 9000\tLoss: 64338.49219\tPPL: 8.78727\tbleu: 14.40652\tLR: 0.00030000\t*\r\n",
      "Steps: 10000\tLoss: 62378.29688\tPPL: 8.22428\tbleu: 15.51252\tLR: 0.00030000\t*\r\n",
      "Steps: 11000\tLoss: 61080.13672\tPPL: 7.87143\tbleu: 15.65950\tLR: 0.00030000\t*\r\n",
      "Steps: 12000\tLoss: 59583.07812\tPPL: 7.48327\tbleu: 16.62673\tLR: 0.00030000\t*\r\n",
      "Steps: 13000\tLoss: 58631.89453\tPPL: 7.24665\tbleu: 16.84451\tLR: 0.00030000\t*\r\n",
      "Steps: 14000\tLoss: 57596.15625\tPPL: 6.99750\tbleu: 17.81381\tLR: 0.00030000\t*\r\n",
      "Steps: 15000\tLoss: 56781.50781\tPPL: 6.80757\tbleu: 18.55316\tLR: 0.00030000\t*\r\n",
      "Steps: 16000\tLoss: 55835.71484\tPPL: 6.59352\tbleu: 18.55196\tLR: 0.00030000\t*\r\n",
      "Steps: 17000\tLoss: 55276.61719\tPPL: 6.47016\tbleu: 18.45183\tLR: 0.00030000\t*\r\n",
      "Steps: 18000\tLoss: 54626.72656\tPPL: 6.32967\tbleu: 19.17805\tLR: 0.00030000\t*\r\n",
      "Steps: 19000\tLoss: 53792.41797\tPPL: 6.15378\tbleu: 19.29552\tLR: 0.00030000\t*\r\n",
      "Steps: 20000\tLoss: 53332.37891\tPPL: 6.05889\tbleu: 19.92834\tLR: 0.00030000\t*\r\n",
      "Steps: 21000\tLoss: 52872.02734\tPPL: 5.96540\tbleu: 20.49299\tLR: 0.00030000\t*\r\n",
      "Steps: 22000\tLoss: 52241.17188\tPPL: 5.83962\tbleu: 20.58551\tLR: 0.00030000\t*\r\n",
      "Steps: 23000\tLoss: 52303.13281\tPPL: 5.85186\tbleu: 20.22862\tLR: 0.00030000\t\r\n",
      "Steps: 24000\tLoss: 51739.83984\tPPL: 5.74156\tbleu: 19.92725\tLR: 0.00030000\t*\r\n",
      "Steps: 25000\tLoss: 51244.48047\tPPL: 5.64629\tbleu: 21.47820\tLR: 0.00030000\t*\r\n",
      "Steps: 26000\tLoss: 51034.22266\tPPL: 5.60633\tbleu: 21.42490\tLR: 0.00030000\t*\r\n",
      "Steps: 27000\tLoss: 50560.17969\tPPL: 5.51727\tbleu: 21.23749\tLR: 0.00030000\t*\r\n",
      "Steps: 28000\tLoss: 50354.13672\tPPL: 5.47900\tbleu: 22.25879\tLR: 0.00030000\t*\r\n",
      "Steps: 29000\tLoss: 49823.43750\tPPL: 5.38166\tbleu: 21.91875\tLR: 0.00030000\t*\r\n",
      "Steps: 30000\tLoss: 49705.75391\tPPL: 5.36031\tbleu: 22.65837\tLR: 0.00030000\t*\r\n",
      "Steps: 31000\tLoss: 49343.94141\tPPL: 5.29519\tbleu: 21.83886\tLR: 0.00030000\t*\r\n",
      "Steps: 32000\tLoss: 49154.03516\tPPL: 5.26134\tbleu: 22.67573\tLR: 0.00030000\t*\r\n",
      "Steps: 33000\tLoss: 49051.19922\tPPL: 5.24309\tbleu: 21.96634\tLR: 0.00030000\t*\r\n",
      "Steps: 34000\tLoss: 48749.59766\tPPL: 5.18995\tbleu: 22.94899\tLR: 0.00030000\t*\r\n",
      "Steps: 35000\tLoss: 48539.97266\tPPL: 5.15333\tbleu: 23.56121\tLR: 0.00030000\t*\r\n",
      "Steps: 36000\tLoss: 48129.71484\tPPL: 5.08240\tbleu: 23.03337\tLR: 0.00030000\t*\r\n",
      "Steps: 37000\tLoss: 47975.33984\tPPL: 5.05597\tbleu: 23.29198\tLR: 0.00030000\t*\r\n",
      "Steps: 38000\tLoss: 47820.10156\tPPL: 5.02953\tbleu: 23.62602\tLR: 0.00030000\t*\r\n",
      "Steps: 39000\tLoss: 47529.84375\tPPL: 4.98045\tbleu: 23.61620\tLR: 0.00030000\t*\r\n",
      "Steps: 40000\tLoss: 47423.77344\tPPL: 4.96264\tbleu: 23.45527\tLR: 0.00030000\t*\r\n",
      "Steps: 41000\tLoss: 47236.06641\tPPL: 4.93127\tbleu: 23.62694\tLR: 0.00030000\t*\r\n",
      "Steps: 42000\tLoss: 47692.08984\tPPL: 5.00782\tbleu: 23.82759\tLR: 0.00030000\t\r\n",
      "Steps: 43000\tLoss: 46881.66016\tPPL: 4.87259\tbleu: 24.27276\tLR: 0.00030000\t*\r\n",
      "Steps: 44000\tLoss: 46783.26172\tPPL: 4.85642\tbleu: 23.78068\tLR: 0.00030000\t*\r\n",
      "Steps: 45000\tLoss: 46635.38672\tPPL: 4.83222\tbleu: 24.12497\tLR: 0.00030000\t*\r\n",
      "Steps: 46000\tLoss: 46367.61719\tPPL: 4.78871\tbleu: 24.22380\tLR: 0.00030000\t*\r\n",
      "Steps: 47000\tLoss: 46688.40234\tPPL: 4.84089\tbleu: 24.35578\tLR: 0.00030000\t\r\n",
      "Steps: 48000\tLoss: 46406.66406\tPPL: 4.79503\tbleu: 23.99385\tLR: 0.00030000\t\r\n",
      "Steps: 49000\tLoss: 45963.84375\tPPL: 4.72384\tbleu: 24.58960\tLR: 0.00030000\t*\r\n",
      "Steps: 50000\tLoss: 46006.55859\tPPL: 4.73066\tbleu: 24.77198\tLR: 0.00030000\t\r\n",
      "Steps: 51000\tLoss: 45738.01953\tPPL: 4.68795\tbleu: 24.89775\tLR: 0.00030000\t*\r\n",
      "Steps: 52000\tLoss: 45752.34375\tPPL: 4.69022\tbleu: 23.99863\tLR: 0.00030000\t\r\n",
      "Steps: 53000\tLoss: 45526.42969\tPPL: 4.65456\tbleu: 24.74591\tLR: 0.00030000\t*\r\n",
      "Steps: 54000\tLoss: 45278.15234\tPPL: 4.61569\tbleu: 24.67634\tLR: 0.00030000\t*\r\n",
      "Steps: 55000\tLoss: 45355.69922\tPPL: 4.62779\tbleu: 24.65326\tLR: 0.00030000\t\r\n",
      "Steps: 56000\tLoss: 45490.61328\tPPL: 4.64893\tbleu: 25.09611\tLR: 0.00030000\t\r\n",
      "Steps: 57000\tLoss: 45200.48047\tPPL: 4.60359\tbleu: 25.22133\tLR: 0.00030000\t*\r\n",
      "Steps: 58000\tLoss: 44995.55469\tPPL: 4.57184\tbleu: 25.29932\tLR: 0.00030000\t*\r\n",
      "Steps: 59000\tLoss: 44955.99609\tPPL: 4.56573\tbleu: 25.46439\tLR: 0.00030000\t*\r\n",
      "Steps: 60000\tLoss: 44820.85156\tPPL: 4.54493\tbleu: 25.20613\tLR: 0.00030000\t*\r\n",
      "Steps: 61000\tLoss: 44636.42969\tPPL: 4.51671\tbleu: 25.55002\tLR: 0.00030000\t*\r\n",
      "Steps: 62000\tLoss: 44655.56250\tPPL: 4.51963\tbleu: 25.27109\tLR: 0.00030000\t\r\n",
      "Steps: 63000\tLoss: 44544.30469\tPPL: 4.50268\tbleu: 25.55201\tLR: 0.00030000\t*\r\n",
      "Steps: 64000\tLoss: 44508.72656\tPPL: 4.49727\tbleu: 25.26382\tLR: 0.00030000\t*\r\n",
      "Steps: 65000\tLoss: 44498.60547\tPPL: 4.49573\tbleu: 25.35637\tLR: 0.00030000\t*\r\n",
      "Steps: 66000\tLoss: 44215.51172\tPPL: 4.45294\tbleu: 25.86049\tLR: 0.00030000\t*\r\n",
      "Steps: 67000\tLoss: 44074.87109\tPPL: 4.43184\tbleu: 25.79971\tLR: 0.00030000\t*\r\n",
      "Steps: 68000\tLoss: 44164.09375\tPPL: 4.44522\tbleu: 25.77768\tLR: 0.00030000\t\r\n",
      "Steps: 69000\tLoss: 44289.98828\tPPL: 4.46416\tbleu: 25.70074\tLR: 0.00030000\t\r\n",
      "Steps: 70000\tLoss: 44060.89062\tPPL: 4.42975\tbleu: 25.51240\tLR: 0.00030000\t*\r\n",
      "Steps: 71000\tLoss: 43942.82812\tPPL: 4.41212\tbleu: 25.96262\tLR: 0.00030000\t*\r\n",
      "Steps: 72000\tLoss: 43766.09375\tPPL: 4.38585\tbleu: 25.94004\tLR: 0.00030000\t*\r\n",
      "Steps: 73000\tLoss: 43722.92188\tPPL: 4.37946\tbleu: 25.86571\tLR: 0.00030000\t*\r\n",
      "Steps: 74000\tLoss: 43771.98828\tPPL: 4.38673\tbleu: 25.91968\tLR: 0.00030000\t\r\n",
      "Steps: 75000\tLoss: 43686.18359\tPPL: 4.37403\tbleu: 25.98464\tLR: 0.00030000\t*\r\n",
      "Steps: 76000\tLoss: 43600.23828\tPPL: 4.36135\tbleu: 25.95412\tLR: 0.00030000\t*\r\n",
      "Steps: 77000\tLoss: 43480.37500\tPPL: 4.34373\tbleu: 26.10838\tLR: 0.00030000\t*\r\n",
      "Steps: 78000\tLoss: 43437.30859\tPPL: 4.33741\tbleu: 26.01543\tLR: 0.00030000\t*\r\n",
      "Steps: 79000\tLoss: 43381.65234\tPPL: 4.32927\tbleu: 25.90944\tLR: 0.00030000\t*\r\n",
      "Steps: 80000\tLoss: 43189.37500\tPPL: 4.30124\tbleu: 26.71502\tLR: 0.00030000\t*\r\n",
      "Steps: 81000\tLoss: 43206.65234\tPPL: 4.30375\tbleu: 26.58806\tLR: 0.00030000\t\r\n",
      "Steps: 82000\tLoss: 43276.73047\tPPL: 4.31395\tbleu: 26.39735\tLR: 0.00030000\t\r\n",
      "Steps: 83000\tLoss: 43271.87109\tPPL: 4.31324\tbleu: 27.05813\tLR: 0.00030000\t\r\n",
      "Steps: 84000\tLoss: 43082.50391\tPPL: 4.28574\tbleu: 26.34011\tLR: 0.00030000\t*\r\n",
      "Steps: 85000\tLoss: 43082.57031\tPPL: 4.28575\tbleu: 26.34041\tLR: 0.00030000\t\r\n",
      "Steps: 86000\tLoss: 43508.96484\tPPL: 4.34793\tbleu: 26.07952\tLR: 0.00030000\t\r\n",
      "Steps: 87000\tLoss: 42968.94141\tPPL: 4.26933\tbleu: 26.71034\tLR: 0.00030000\t*\r\n",
      "Steps: 88000\tLoss: 42866.58984\tPPL: 4.25460\tbleu: 27.23061\tLR: 0.00030000\t*\r\n",
      "Steps: 89000\tLoss: 42920.01953\tPPL: 4.26228\tbleu: 27.20530\tLR: 0.00030000\t\r\n",
      "Steps: 90000\tLoss: 42742.44531\tPPL: 4.23679\tbleu: 26.99235\tLR: 0.00030000\t*\r\n",
      "Steps: 91000\tLoss: 42802.66406\tPPL: 4.24542\tbleu: 26.99323\tLR: 0.00030000\t\r\n",
      "Steps: 92000\tLoss: 42720.53516\tPPL: 4.23366\tbleu: 27.13692\tLR: 0.00030000\t*\r\n",
      "Steps: 93000\tLoss: 42736.42188\tPPL: 4.23593\tbleu: 27.21527\tLR: 0.00030000\t\r\n",
      "Steps: 94000\tLoss: 42732.86719\tPPL: 4.23542\tbleu: 27.09541\tLR: 0.00030000\t\r\n",
      "Steps: 95000\tLoss: 42683.00391\tPPL: 4.22829\tbleu: 27.43607\tLR: 0.00030000\t*\r\n",
      "Steps: 96000\tLoss: 42779.67969\tPPL: 4.24212\tbleu: 26.76358\tLR: 0.00030000\t\r\n",
      "Steps: 97000\tLoss: 42734.20703\tPPL: 4.23561\tbleu: 26.82313\tLR: 0.00030000\t\r\n",
      "Steps: 98000\tLoss: 42487.83984\tPPL: 4.20051\tbleu: 26.81564\tLR: 0.00030000\t*\r\n",
      "Steps: 99000\tLoss: 42561.01953\tPPL: 4.21091\tbleu: 27.25598\tLR: 0.00030000\t\r\n",
      "Steps: 100000\tLoss: 42515.51172\tPPL: 4.20444\tbleu: 27.02230\tLR: 0.00030000\t\r\n",
      "Steps: 101000\tLoss: 42570.89844\tPPL: 4.21231\tbleu: 26.97419\tLR: 0.00030000\t\r\n",
      "Steps: 102000\tLoss: 42458.23438\tPPL: 4.19631\tbleu: 26.98413\tLR: 0.00030000\t*\r\n",
      "Steps: 103000\tLoss: 42256.05859\tPPL: 4.16775\tbleu: 27.01413\tLR: 0.00030000\t*\r\n",
      "Steps: 104000\tLoss: 42217.95312\tPPL: 4.16239\tbleu: 26.97259\tLR: 0.00030000\t*\r\n",
      "Steps: 105000\tLoss: 42309.16016\tPPL: 4.17523\tbleu: 27.15736\tLR: 0.00030000\t\r\n",
      "Steps: 106000\tLoss: 42175.64453\tPPL: 4.15645\tbleu: 27.27136\tLR: 0.00030000\t*\r\n",
      "Steps: 107000\tLoss: 42117.47266\tPPL: 4.14829\tbleu: 27.40689\tLR: 0.00030000\t*\r\n",
      "Steps: 108000\tLoss: 42154.33203\tPPL: 4.15345\tbleu: 27.13642\tLR: 0.00030000\t\r\n",
      "Steps: 109000\tLoss: 42130.86328\tPPL: 4.15016\tbleu: 27.50679\tLR: 0.00030000\t\r\n",
      "Steps: 110000\tLoss: 42108.83594\tPPL: 4.14708\tbleu: 27.49709\tLR: 0.00030000\t*\r\n",
      "Steps: 111000\tLoss: 42005.95312\tPPL: 4.13269\tbleu: 27.53611\tLR: 0.00030000\t*\r\n",
      "Steps: 112000\tLoss: 42125.15234\tPPL: 4.14936\tbleu: 27.67555\tLR: 0.00030000\t\r\n",
      "Steps: 113000\tLoss: 41784.39844\tPPL: 4.10188\tbleu: 26.89100\tLR: 0.00030000\t*\r\n",
      "Steps: 114000\tLoss: 41840.89453\tPPL: 4.10971\tbleu: 27.86662\tLR: 0.00030000\t\r\n",
      "Steps: 115000\tLoss: 41749.22266\tPPL: 4.09700\tbleu: 27.88800\tLR: 0.00030000\t*\r\n",
      "Steps: 116000\tLoss: 41770.01562\tPPL: 4.09988\tbleu: 27.39544\tLR: 0.00030000\t\r\n",
      "Steps: 117000\tLoss: 41847.34766\tPPL: 4.11061\tbleu: 27.54739\tLR: 0.00030000\t\r\n",
      "Steps: 118000\tLoss: 41751.44141\tPPL: 4.09731\tbleu: 27.57260\tLR: 0.00030000\t\r\n",
      "Steps: 119000\tLoss: 41646.80078\tPPL: 4.08285\tbleu: 27.58578\tLR: 0.00030000\t*\r\n",
      "Steps: 120000\tLoss: 41766.10547\tPPL: 4.09934\tbleu: 27.51823\tLR: 0.00030000\t\r\n",
      "Steps: 121000\tLoss: 41909.54688\tPPL: 4.11925\tbleu: 27.22325\tLR: 0.00030000\t\r\n",
      "Steps: 122000\tLoss: 41763.15625\tPPL: 4.09893\tbleu: 27.72097\tLR: 0.00030000\t\r\n",
      "Steps: 123000\tLoss: 41713.44922\tPPL: 4.09206\tbleu: 27.27351\tLR: 0.00030000\t\r\n",
      "Steps: 124000\tLoss: 41700.37500\tPPL: 4.09025\tbleu: 27.44044\tLR: 0.00030000\t\r\n",
      "Steps: 125000\tLoss: 41533.00391\tPPL: 4.06719\tbleu: 27.21652\tLR: 0.00030000\t*\r\n",
      "Steps: 126000\tLoss: 41666.35156\tPPL: 4.08555\tbleu: 27.49333\tLR: 0.00030000\t\r\n",
      "Steps: 127000\tLoss: 41536.27734\tPPL: 4.06764\tbleu: 27.59551\tLR: 0.00030000\t\r\n",
      "Steps: 128000\tLoss: 41669.67188\tPPL: 4.08601\tbleu: 27.86542\tLR: 0.00030000\t\r\n",
      "Steps: 129000\tLoss: 41558.97656\tPPL: 4.07076\tbleu: 28.03323\tLR: 0.00030000\t\r\n",
      "Steps: 130000\tLoss: 41450.28516\tPPL: 4.05584\tbleu: 27.81447\tLR: 0.00030000\t*\r\n",
      "Steps: 131000\tLoss: 41441.75391\tPPL: 4.05467\tbleu: 27.89550\tLR: 0.00030000\t*\r\n",
      "Steps: 132000\tLoss: 41601.04688\tPPL: 4.07655\tbleu: 27.83780\tLR: 0.00030000\t\r\n",
      "Steps: 133000\tLoss: 41597.44141\tPPL: 4.07605\tbleu: 27.64847\tLR: 0.00030000\t\r\n",
      "Steps: 134000\tLoss: 41431.37891\tPPL: 4.05325\tbleu: 28.10347\tLR: 0.00030000\t*\r\n",
      "Steps: 135000\tLoss: 41287.89062\tPPL: 4.03365\tbleu: 28.30537\tLR: 0.00030000\t*\r\n",
      "Steps: 136000\tLoss: 41520.46875\tPPL: 4.06547\tbleu: 28.03441\tLR: 0.00030000\t\r\n",
      "Steps: 137000\tLoss: 41284.65625\tPPL: 4.03321\tbleu: 28.14462\tLR: 0.00030000\t*\r\n",
      "Steps: 138000\tLoss: 41269.48828\tPPL: 4.03115\tbleu: 28.39196\tLR: 0.00030000\t*\r\n",
      "Steps: 139000\tLoss: 41243.76953\tPPL: 4.02765\tbleu: 27.60392\tLR: 0.00030000\t*\r\n",
      "Steps: 140000\tLoss: 41236.44141\tPPL: 4.02665\tbleu: 27.79399\tLR: 0.00030000\t*\r\n",
      "Steps: 141000\tLoss: 41259.14453\tPPL: 4.02974\tbleu: 28.28609\tLR: 0.00030000\t\r\n",
      "Steps: 142000\tLoss: 41258.64844\tPPL: 4.02967\tbleu: 28.13920\tLR: 0.00030000\t\r\n",
      "Steps: 143000\tLoss: 41192.26953\tPPL: 4.02065\tbleu: 28.19409\tLR: 0.00030000\t*\r\n",
      "Steps: 144000\tLoss: 41174.01953\tPPL: 4.01817\tbleu: 28.03398\tLR: 0.00030000\t*\r\n",
      "Steps: 145000\tLoss: 41079.01562\tPPL: 4.00529\tbleu: 28.48257\tLR: 0.00030000\t*\r\n",
      "Steps: 146000\tLoss: 41082.60547\tPPL: 4.00578\tbleu: 28.28648\tLR: 0.00030000\t\r\n",
      "Steps: 147000\tLoss: 41109.38672\tPPL: 4.00941\tbleu: 27.89879\tLR: 0.00030000\t\r\n",
      "Steps: 148000\tLoss: 41156.06641\tPPL: 4.01573\tbleu: 27.85472\tLR: 0.00030000\t\r\n",
      "Steps: 149000\tLoss: 40998.12109\tPPL: 3.99436\tbleu: 28.14334\tLR: 0.00030000\t*\r\n",
      "Steps: 150000\tLoss: 40995.73047\tPPL: 3.99404\tbleu: 27.90726\tLR: 0.00030000\t*\r\n",
      "Steps: 151000\tLoss: 41003.54297\tPPL: 3.99510\tbleu: 28.07529\tLR: 0.00030000\t\r\n",
      "Steps: 152000\tLoss: 40997.42969\tPPL: 3.99427\tbleu: 27.92925\tLR: 0.00030000\t\r\n",
      "Steps: 153000\tLoss: 40971.52734\tPPL: 3.99078\tbleu: 28.01540\tLR: 0.00030000\t*\r\n",
      "Steps: 154000\tLoss: 40796.38672\tPPL: 3.96724\tbleu: 28.04420\tLR: 0.00030000\t*\r\n",
      "Steps: 155000\tLoss: 41038.88281\tPPL: 3.99987\tbleu: 27.87313\tLR: 0.00030000\t\r\n",
      "Steps: 156000\tLoss: 40912.42578\tPPL: 3.98282\tbleu: 28.26356\tLR: 0.00030000\t\r\n",
      "Steps: 157000\tLoss: 40818.16016\tPPL: 3.97016\tbleu: 28.19387\tLR: 0.00030000\t\r\n",
      "Steps: 158000\tLoss: 40989.26562\tPPL: 3.99317\tbleu: 28.03864\tLR: 0.00030000\t\r\n",
      "Steps: 159000\tLoss: 40661.41016\tPPL: 3.94919\tbleu: 28.24657\tLR: 0.00030000\t*\r\n",
      "Steps: 160000\tLoss: 40752.80859\tPPL: 3.96140\tbleu: 28.59965\tLR: 0.00030000\t\r\n",
      "Steps: 161000\tLoss: 40616.09766\tPPL: 3.94315\tbleu: 28.56946\tLR: 0.00030000\t*\r\n",
      "Steps: 162000\tLoss: 40671.16406\tPPL: 3.95049\tbleu: 28.84163\tLR: 0.00030000\t\r\n",
      "Steps: 163000\tLoss: 40756.37891\tPPL: 3.96188\tbleu: 28.42171\tLR: 0.00030000\t\r\n",
      "Steps: 164000\tLoss: 40605.39844\tPPL: 3.94173\tbleu: 28.60857\tLR: 0.00030000\t*\r\n",
      "Steps: 165000\tLoss: 40696.53125\tPPL: 3.95388\tbleu: 28.36107\tLR: 0.00030000\t\r\n",
      "Steps: 166000\tLoss: 40709.00000\tPPL: 3.95554\tbleu: 28.39606\tLR: 0.00030000\t\r\n",
      "Steps: 167000\tLoss: 40678.64062\tPPL: 3.95149\tbleu: 28.46372\tLR: 0.00030000\t\r\n",
      "Steps: 168000\tLoss: 40764.42188\tPPL: 3.96296\tbleu: 28.70552\tLR: 0.00030000\t\r\n",
      "Steps: 169000\tLoss: 40690.58594\tPPL: 3.95308\tbleu: 29.16179\tLR: 0.00030000\t\r\n",
      "Steps: 170000\tLoss: 40766.20703\tPPL: 3.96320\tbleu: 28.13120\tLR: 0.00021000\t\r\n",
      "Steps: 171000\tLoss: 40408.37109\tPPL: 3.91558\tbleu: 28.70033\tLR: 0.00021000\t*\r\n",
      "Steps: 172000\tLoss: 40373.54297\tPPL: 3.91097\tbleu: 28.21079\tLR: 0.00021000\t*\r\n",
      "Steps: 173000\tLoss: 40381.03516\tPPL: 3.91196\tbleu: 28.48932\tLR: 0.00021000\t\r\n",
      "Steps: 174000\tLoss: 40309.50391\tPPL: 3.90252\tbleu: 28.96801\tLR: 0.00021000\t*\r\n",
      "Steps: 175000\tLoss: 40399.07031\tPPL: 3.91435\tbleu: 28.55233\tLR: 0.00021000\t\r\n",
      "Steps: 176000\tLoss: 40274.07422\tPPL: 3.89786\tbleu: 28.59915\tLR: 0.00021000\t*\r\n",
      "Steps: 177000\tLoss: 40310.02734\tPPL: 3.90259\tbleu: 28.58224\tLR: 0.00021000\t\r\n",
      "Steps: 178000\tLoss: 40242.07031\tPPL: 3.89364\tbleu: 28.53344\tLR: 0.00021000\t*\r\n",
      "Steps: 179000\tLoss: 40261.34375\tPPL: 3.89618\tbleu: 28.74306\tLR: 0.00021000\t\r\n",
      "Steps: 180000\tLoss: 40359.41797\tPPL: 3.90911\tbleu: 28.84275\tLR: 0.00021000\t\r\n",
      "Steps: 181000\tLoss: 40174.29297\tPPL: 3.88474\tbleu: 28.94268\tLR: 0.00021000\t*\r\n",
      "Steps: 182000\tLoss: 40188.29297\tPPL: 3.88658\tbleu: 28.79410\tLR: 0.00021000\t\r\n",
      "Steps: 183000\tLoss: 40179.22266\tPPL: 3.88539\tbleu: 28.89476\tLR: 0.00021000\t\r\n",
      "Steps: 184000\tLoss: 40014.63672\tPPL: 3.86385\tbleu: 29.11380\tLR: 0.00021000\t*\r\n",
      "Steps: 185000\tLoss: 40172.47656\tPPL: 3.88450\tbleu: 29.13836\tLR: 0.00021000\t\r\n",
      "Steps: 186000\tLoss: 40009.59375\tPPL: 3.86319\tbleu: 29.34377\tLR: 0.00021000\t*\r\n",
      "Steps: 187000\tLoss: 40115.84766\tPPL: 3.87708\tbleu: 29.26508\tLR: 0.00021000\t\r\n",
      "Steps: 188000\tLoss: 40050.41797\tPPL: 3.86852\tbleu: 29.10378\tLR: 0.00021000\t\r\n",
      "Steps: 189000\tLoss: 40070.10156\tPPL: 3.87109\tbleu: 28.98482\tLR: 0.00021000\t\r\n",
      "Steps: 190000\tLoss: 40000.05469\tPPL: 3.86194\tbleu: 29.04826\tLR: 0.00021000\t*\r\n",
      "Steps: 191000\tLoss: 40065.00391\tPPL: 3.87043\tbleu: 29.09903\tLR: 0.00021000\t\r\n",
      "Steps: 192000\tLoss: 40046.81641\tPPL: 3.86805\tbleu: 29.31513\tLR: 0.00021000\t\r\n",
      "Steps: 193000\tLoss: 40107.01562\tPPL: 3.87592\tbleu: 28.72916\tLR: 0.00021000\t\r\n",
      "Steps: 194000\tLoss: 39939.77344\tPPL: 3.85409\tbleu: 28.77790\tLR: 0.00021000\t*\r\n",
      "Steps: 195000\tLoss: 39999.35547\tPPL: 3.86185\tbleu: 29.18863\tLR: 0.00021000\t\r\n",
      "Steps: 196000\tLoss: 40041.59375\tPPL: 3.86737\tbleu: 28.97678\tLR: 0.00021000\t\r\n",
      "Steps: 197000\tLoss: 40091.58594\tPPL: 3.87390\tbleu: 29.10821\tLR: 0.00021000\t\r\n",
      "Steps: 198000\tLoss: 40072.01953\tPPL: 3.87134\tbleu: 28.69107\tLR: 0.00021000\t\r\n",
      "Steps: 199000\tLoss: 39961.18359\tPPL: 3.85688\tbleu: 28.89230\tLR: 0.00021000\t\r\n",
      "Steps: 200000\tLoss: 39941.57422\tPPL: 3.85432\tbleu: 28.69717\tLR: 0.00014700\t\r\n",
      "Steps: 201000\tLoss: 39719.39062\tPPL: 3.82550\tbleu: 29.07003\tLR: 0.00014700\t*\r\n",
      "Steps: 202000\tLoss: 39728.58984\tPPL: 3.82669\tbleu: 29.29578\tLR: 0.00014700\t\r\n",
      "Steps: 203000\tLoss: 39766.37891\tPPL: 3.83158\tbleu: 29.24292\tLR: 0.00014700\t\r\n",
      "Steps: 204000\tLoss: 39757.53516\tPPL: 3.83044\tbleu: 28.98475\tLR: 0.00014700\t\r\n",
      "Steps: 205000\tLoss: 39756.62109\tPPL: 3.83032\tbleu: 29.43822\tLR: 0.00014700\t\r\n",
      "Steps: 206000\tLoss: 39728.87500\tPPL: 3.82673\tbleu: 29.31968\tLR: 0.00014700\t\r\n",
      "Steps: 207000\tLoss: 39804.19531\tPPL: 3.83648\tbleu: 29.10816\tLR: 0.00010290\t\r\n",
      "Steps: 208000\tLoss: 39632.53125\tPPL: 3.81430\tbleu: 29.56499\tLR: 0.00010290\t*\r\n",
      "Steps: 209000\tLoss: 39577.47266\tPPL: 3.80721\tbleu: 29.23017\tLR: 0.00010290\t*\r\n",
      "Steps: 210000\tLoss: 39549.51172\tPPL: 3.80361\tbleu: 29.43340\tLR: 0.00010290\t*\r\n",
      "Steps: 211000\tLoss: 39553.62109\tPPL: 3.80414\tbleu: 29.45998\tLR: 0.00010290\t\r\n",
      "Steps: 212000\tLoss: 39641.36328\tPPL: 3.81543\tbleu: 29.12107\tLR: 0.00010290\t\r\n",
      "Steps: 213000\tLoss: 39588.56641\tPPL: 3.80863\tbleu: 29.56784\tLR: 0.00010290\t\r\n",
      "Steps: 214000\tLoss: 39608.98438\tPPL: 3.81126\tbleu: 29.42443\tLR: 0.00010290\t\r\n",
      "Steps: 215000\tLoss: 39584.89844\tPPL: 3.80816\tbleu: 29.39054\tLR: 0.00010290\t\r\n",
      "Steps: 216000\tLoss: 39467.81250\tPPL: 3.79313\tbleu: 29.67875\tLR: 0.00010290\t*\r\n",
      "Steps: 217000\tLoss: 39499.17969\tPPL: 3.79715\tbleu: 29.50963\tLR: 0.00010290\t\r\n",
      "Steps: 218000\tLoss: 39446.57422\tPPL: 3.79041\tbleu: 29.86833\tLR: 0.00010290\t*\r\n",
      "Steps: 219000\tLoss: 39518.99219\tPPL: 3.79969\tbleu: 29.52752\tLR: 0.00010290\t\r\n",
      "Steps: 220000\tLoss: 39496.85156\tPPL: 3.79685\tbleu: 29.58048\tLR: 0.00010290\t\r\n",
      "Steps: 221000\tLoss: 39549.53906\tPPL: 3.80362\tbleu: 29.23343\tLR: 0.00010290\t\r\n",
      "Steps: 222000\tLoss: 39497.55469\tPPL: 3.79694\tbleu: 29.38254\tLR: 0.00010290\t\r\n",
      "Steps: 223000\tLoss: 39435.68750\tPPL: 3.78902\tbleu: 29.58638\tLR: 0.00010290\t*\r\n",
      "Steps: 224000\tLoss: 39519.30078\tPPL: 3.79973\tbleu: 29.57605\tLR: 0.00010290\t\r\n",
      "Steps: 225000\tLoss: 39526.18359\tPPL: 3.80062\tbleu: 29.45001\tLR: 0.00010290\t\r\n",
      "Steps: 226000\tLoss: 39384.35938\tPPL: 3.78245\tbleu: 29.29467\tLR: 0.00010290\t*\r\n",
      "Steps: 227000\tLoss: 39556.02344\tPPL: 3.80445\tbleu: 29.58378\tLR: 0.00010290\t\r\n",
      "Steps: 228000\tLoss: 39437.68359\tPPL: 3.78927\tbleu: 29.53031\tLR: 0.00010290\t\r\n",
      "Steps: 229000\tLoss: 39496.60156\tPPL: 3.79682\tbleu: 29.56506\tLR: 0.00010290\t\r\n",
      "Steps: 230000\tLoss: 39400.94922\tPPL: 3.78457\tbleu: 29.44868\tLR: 0.00010290\t\r\n",
      "Steps: 231000\tLoss: 39428.58594\tPPL: 3.78811\tbleu: 29.71520\tLR: 0.00010290\t\r\n",
      "Steps: 232000\tLoss: 39418.94531\tPPL: 3.78688\tbleu: 29.59938\tLR: 0.00007203\t\r\n",
      "Steps: 233000\tLoss: 39356.26172\tPPL: 3.77887\tbleu: 29.71300\tLR: 0.00007203\t*\r\n",
      "Steps: 234000\tLoss: 39363.27734\tPPL: 3.77976\tbleu: 29.85371\tLR: 0.00007203\t\r\n",
      "Steps: 235000\tLoss: 39316.39844\tPPL: 3.77378\tbleu: 29.51021\tLR: 0.00007203\t*\r\n",
      "Steps: 236000\tLoss: 39352.76172\tPPL: 3.77842\tbleu: 29.72986\tLR: 0.00007203\t\r\n",
      "Steps: 237000\tLoss: 39366.67188\tPPL: 3.78019\tbleu: 29.61093\tLR: 0.00007203\t\r\n",
      "Steps: 238000\tLoss: 39324.51562\tPPL: 3.77482\tbleu: 29.51633\tLR: 0.00007203\t\r\n",
      "Steps: 239000\tLoss: 39349.10547\tPPL: 3.77795\tbleu: 29.73000\tLR: 0.00007203\t\r\n",
      "Steps: 240000\tLoss: 39358.99609\tPPL: 3.77921\tbleu: 29.85234\tLR: 0.00007203\t\r\n",
      "Steps: 241000\tLoss: 39282.72266\tPPL: 3.76949\tbleu: 29.71081\tLR: 0.00007203\t*\r\n",
      "Steps: 242000\tLoss: 39312.30078\tPPL: 3.77326\tbleu: 29.60884\tLR: 0.00007203\t\r\n",
      "Steps: 243000\tLoss: 39287.11719\tPPL: 3.77005\tbleu: 29.94617\tLR: 0.00007203\t\r\n",
      "Steps: 244000\tLoss: 39349.08203\tPPL: 3.77795\tbleu: 29.80512\tLR: 0.00007203\t\r\n",
      "Steps: 245000\tLoss: 39303.03125\tPPL: 3.77208\tbleu: 29.81018\tLR: 0.00007203\t\r\n",
      "Steps: 246000\tLoss: 39357.03516\tPPL: 3.77896\tbleu: 29.62981\tLR: 0.00007203\t\r\n",
      "Steps: 247000\tLoss: 39245.72266\tPPL: 3.76478\tbleu: 29.81336\tLR: 0.00007203\t*\r\n",
      "Steps: 248000\tLoss: 39282.59766\tPPL: 3.76947\tbleu: 29.81455\tLR: 0.00007203\t\r\n",
      "Steps: 249000\tLoss: 39322.32812\tPPL: 3.77454\tbleu: 29.63206\tLR: 0.00007203\t\r\n",
      "Steps: 250000\tLoss: 39273.95312\tPPL: 3.76837\tbleu: 29.54439\tLR: 0.00007203\t\r\n",
      "Steps: 251000\tLoss: 39319.13281\tPPL: 3.77413\tbleu: 29.87973\tLR: 0.00007203\t\r\n",
      "Steps: 252000\tLoss: 39307.85547\tPPL: 3.77269\tbleu: 29.65472\tLR: 0.00007203\t\r\n",
      "Steps: 253000\tLoss: 39240.53906\tPPL: 3.76412\tbleu: 29.64379\tLR: 0.00007203\t*\r\n",
      "Steps: 254000\tLoss: 39213.77734\tPPL: 3.76072\tbleu: 29.66877\tLR: 0.00007203\t*\r\n",
      "Steps: 255000\tLoss: 39209.02344\tPPL: 3.76012\tbleu: 29.49227\tLR: 0.00007203\t*\r\n",
      "Steps: 256000\tLoss: 39281.69141\tPPL: 3.76936\tbleu: 29.78438\tLR: 0.00007203\t\r\n",
      "Steps: 257000\tLoss: 39316.33984\tPPL: 3.77377\tbleu: 29.73665\tLR: 0.00007203\t\r\n",
      "Steps: 258000\tLoss: 39247.80469\tPPL: 3.76505\tbleu: 29.97161\tLR: 0.00007203\t\r\n",
      "Steps: 259000\tLoss: 39272.75391\tPPL: 3.76822\tbleu: 29.63552\tLR: 0.00007203\t\r\n",
      "Steps: 260000\tLoss: 39201.07812\tPPL: 3.75911\tbleu: 29.80551\tLR: 0.00007203\t*\r\n",
      "Steps: 261000\tLoss: 39299.82422\tPPL: 3.77167\tbleu: 29.76712\tLR: 0.00007203\t\r\n",
      "Steps: 262000\tLoss: 39219.62891\tPPL: 3.76146\tbleu: 29.58390\tLR: 0.00007203\t\r\n",
      "Steps: 263000\tLoss: 39180.19141\tPPL: 3.75646\tbleu: 29.73185\tLR: 0.00007203\t*\r\n",
      "Steps: 264000\tLoss: 39222.64844\tPPL: 3.76185\tbleu: 29.83996\tLR: 0.00007203\t\r\n",
      "Steps: 265000\tLoss: 39203.93359\tPPL: 3.75947\tbleu: 29.71186\tLR: 0.00007203\t\r\n",
      "Steps: 266000\tLoss: 39210.33984\tPPL: 3.76028\tbleu: 29.69321\tLR: 0.00007203\t\r\n",
      "Steps: 267000\tLoss: 39301.10938\tPPL: 3.77183\tbleu: 29.53369\tLR: 0.00007203\t\r\n",
      "Steps: 268000\tLoss: 39208.17969\tPPL: 3.76001\tbleu: 29.47751\tLR: 0.00007203\t\r\n",
      "Steps: 269000\tLoss: 39204.69141\tPPL: 3.75957\tbleu: 29.27165\tLR: 0.00005042\t\r\n",
      "Steps: 270000\tLoss: 39144.94531\tPPL: 3.75199\tbleu: 29.62416\tLR: 0.00005042\t*\r\n",
      "Steps: 271000\tLoss: 39216.93359\tPPL: 3.76112\tbleu: 29.69027\tLR: 0.00005042\t\r\n",
      "Steps: 272000\tLoss: 39148.71094\tPPL: 3.75246\tbleu: 29.93294\tLR: 0.00005042\t\r\n",
      "Steps: 273000\tLoss: 39139.07422\tPPL: 3.75124\tbleu: 29.69409\tLR: 0.00005042\t*\r\n",
      "Steps: 274000\tLoss: 39146.03516\tPPL: 3.75213\tbleu: 29.77171\tLR: 0.00005042\t\r\n",
      "Steps: 275000\tLoss: 39160.29688\tPPL: 3.75393\tbleu: 29.93154\tLR: 0.00005042\t\r\n",
      "Steps: 276000\tLoss: 39120.83594\tPPL: 3.74893\tbleu: 29.69997\tLR: 0.00005042\t*\r\n",
      "Steps: 277000\tLoss: 39159.19922\tPPL: 3.75379\tbleu: 29.70755\tLR: 0.00005042\t\r\n",
      "Steps: 278000\tLoss: 39114.30078\tPPL: 3.74811\tbleu: 29.85794\tLR: 0.00005042\t*\r\n",
      "Steps: 279000\tLoss: 39131.06250\tPPL: 3.75023\tbleu: 29.64314\tLR: 0.00005042\t\r\n",
      "Steps: 280000\tLoss: 39113.30859\tPPL: 3.74798\tbleu: 29.75801\tLR: 0.00005042\t*\r\n",
      "Steps: 281000\tLoss: 39156.24219\tPPL: 3.75342\tbleu: 29.78437\tLR: 0.00005042\t\r\n",
      "Steps: 282000\tLoss: 39150.73047\tPPL: 3.75272\tbleu: 29.77594\tLR: 0.00005042\t\r\n",
      "Steps: 283000\tLoss: 39148.11719\tPPL: 3.75239\tbleu: 29.97633\tLR: 0.00005042\t\r\n",
      "Steps: 284000\tLoss: 39110.62109\tPPL: 3.74764\tbleu: 29.98767\tLR: 0.00005042\t*\r\n",
      "Steps: 285000\tLoss: 39104.16016\tPPL: 3.74682\tbleu: 29.81144\tLR: 0.00005042\t*\r\n",
      "Steps: 286000\tLoss: 39129.65625\tPPL: 3.75005\tbleu: 29.82019\tLR: 0.00005042\t\r\n",
      "Steps: 287000\tLoss: 39094.40234\tPPL: 3.74559\tbleu: 29.99268\tLR: 0.00005042\t*\r\n",
      "Steps: 288000\tLoss: 39131.40625\tPPL: 3.75027\tbleu: 29.74801\tLR: 0.00005042\t\r\n",
      "Steps: 289000\tLoss: 39101.37500\tPPL: 3.74647\tbleu: 29.75216\tLR: 0.00005042\t\r\n",
      "Steps: 290000\tLoss: 39148.58594\tPPL: 3.75245\tbleu: 29.84378\tLR: 0.00005042\t\r\n",
      "Steps: 291000\tLoss: 39169.95703\tPPL: 3.75516\tbleu: 29.96572\tLR: 0.00005042\t\r\n",
      "Steps: 292000\tLoss: 39127.75391\tPPL: 3.74981\tbleu: 29.82250\tLR: 0.00005042\t\r\n",
      "Steps: 293000\tLoss: 39144.92188\tPPL: 3.75198\tbleu: 30.13942\tLR: 0.00003529\t\r\n",
      "Steps: 294000\tLoss: 39060.19141\tPPL: 3.74126\tbleu: 30.23340\tLR: 0.00003529\t*\r\n",
      "Steps: 295000\tLoss: 39106.82031\tPPL: 3.74716\tbleu: 30.19701\tLR: 0.00003529\t\r\n",
      "Steps: 296000\tLoss: 39128.61719\tPPL: 3.74992\tbleu: 30.05731\tLR: 0.00003529\t\r\n",
      "Steps: 297000\tLoss: 39103.60547\tPPL: 3.74675\tbleu: 29.71351\tLR: 0.00003529\t\r\n",
      "Steps: 298000\tLoss: 39077.63672\tPPL: 3.74347\tbleu: 30.03616\tLR: 0.00003529\t\r\n",
      "Steps: 299000\tLoss: 39074.05859\tPPL: 3.74301\tbleu: 30.02235\tLR: 0.00003529\t\r\n",
      "Steps: 300000\tLoss: 39086.00781\tPPL: 3.74453\tbleu: 30.02723\tLR: 0.00002471\t\r\n",
      "Steps: 301000\tLoss: 39041.42188\tPPL: 3.73889\tbleu: 30.05020\tLR: 0.00002471\t*\r\n",
      "Steps: 302000\tLoss: 39071.33594\tPPL: 3.74267\tbleu: 29.93734\tLR: 0.00002471\t\r\n",
      "Steps: 303000\tLoss: 39056.83203\tPPL: 3.74084\tbleu: 29.97613\tLR: 0.00002471\t\r\n",
      "Steps: 304000\tLoss: 39065.11328\tPPL: 3.74188\tbleu: 29.99704\tLR: 0.00002471\t\r\n",
      "Steps: 305000\tLoss: 39079.96875\tPPL: 3.74376\tbleu: 30.08664\tLR: 0.00002471\t\r\n",
      "Steps: 306000\tLoss: 39058.67578\tPPL: 3.74107\tbleu: 30.24091\tLR: 0.00002471\t\r\n",
      "Steps: 307000\tLoss: 39029.03125\tPPL: 3.73733\tbleu: 30.15951\tLR: 0.00002471\t*\r\n",
      "Steps: 308000\tLoss: 39023.77734\tPPL: 3.73666\tbleu: 30.09943\tLR: 0.00002471\t*\r\n",
      "Steps: 309000\tLoss: 39081.64844\tPPL: 3.74397\tbleu: 30.14585\tLR: 0.00002471\t\r\n",
      "Steps: 310000\tLoss: 39058.17578\tPPL: 3.74101\tbleu: 29.90648\tLR: 0.00002471\t\r\n",
      "Steps: 311000\tLoss: 39050.96094\tPPL: 3.74009\tbleu: 29.96362\tLR: 0.00002471\t\r\n",
      "Steps: 312000\tLoss: 39070.39844\tPPL: 3.74255\tbleu: 30.15418\tLR: 0.00002471\t\r\n",
      "Steps: 313000\tLoss: 39074.79297\tPPL: 3.74311\tbleu: 30.15267\tLR: 0.00002471\t\r\n",
      "Steps: 314000\tLoss: 39054.68750\tPPL: 3.74057\tbleu: 30.04901\tLR: 0.00001729\t\r\n",
      "Steps: 315000\tLoss: 39044.05859\tPPL: 3.73922\tbleu: 30.05209\tLR: 0.00001729\t\r\n",
      "Steps: 316000\tLoss: 39043.26562\tPPL: 3.73912\tbleu: 30.12745\tLR: 0.00001729\t\r\n",
      "Steps: 317000\tLoss: 39033.94141\tPPL: 3.73795\tbleu: 30.05006\tLR: 0.00001729\t\r\n",
      "Steps: 318000\tLoss: 39037.84375\tPPL: 3.73844\tbleu: 29.98392\tLR: 0.00001729\t\r\n",
      "Steps: 319000\tLoss: 39045.07812\tPPL: 3.73935\tbleu: 30.06822\tLR: 0.00001729\t\r\n",
      "Steps: 320000\tLoss: 39046.43750\tPPL: 3.73952\tbleu: 30.16860\tLR: 0.00001211\t\r\n",
      "Steps: 321000\tLoss: 39020.66016\tPPL: 3.73627\tbleu: 30.06363\tLR: 0.00001211\t*\r\n",
      "Steps: 322000\tLoss: 39039.70703\tPPL: 3.73867\tbleu: 30.16254\tLR: 0.00001211\t\r\n",
      "Steps: 323000\tLoss: 39015.02734\tPPL: 3.73556\tbleu: 30.24893\tLR: 0.00001211\t*\r\n",
      "Steps: 324000\tLoss: 39021.37500\tPPL: 3.73636\tbleu: 30.16753\tLR: 0.00001211\t\r\n",
      "Steps: 325000\tLoss: 39038.28516\tPPL: 3.73849\tbleu: 30.14093\tLR: 0.00001211\t\r\n",
      "Steps: 326000\tLoss: 39009.57422\tPPL: 3.73487\tbleu: 30.16788\tLR: 0.00001211\t*\r\n",
      "Steps: 327000\tLoss: 39018.38281\tPPL: 3.73598\tbleu: 30.12180\tLR: 0.00001211\t\r\n",
      "Steps: 328000\tLoss: 39023.04297\tPPL: 3.73657\tbleu: 30.14058\tLR: 0.00001211\t\r\n",
      "Steps: 329000\tLoss: 39011.72656\tPPL: 3.73514\tbleu: 30.10501\tLR: 0.00001211\t\r\n",
      "Steps: 330000\tLoss: 39010.86328\tPPL: 3.73503\tbleu: 30.14138\tLR: 0.00001211\t\r\n",
      "Steps: 331000\tLoss: 39013.57422\tPPL: 3.73537\tbleu: 30.13420\tLR: 0.00001211\t\r\n",
      "Steps: 332000\tLoss: 39002.67188\tPPL: 3.73400\tbleu: 30.08330\tLR: 0.00001211\t*\r\n",
      "Steps: 333000\tLoss: 39011.00000\tPPL: 3.73505\tbleu: 29.96446\tLR: 0.00001211\t\r\n",
      "Steps: 334000\tLoss: 38973.82031\tPPL: 3.73036\tbleu: 30.08604\tLR: 0.00001211\t*\r\n",
      "Steps: 335000\tLoss: 38982.62500\tPPL: 3.73147\tbleu: 30.05364\tLR: 0.00001211\t\r\n",
      "Steps: 336000\tLoss: 38997.42578\tPPL: 3.73334\tbleu: 30.20185\tLR: 0.00001211\t\r\n",
      "Steps: 337000\tLoss: 38990.43750\tPPL: 3.73246\tbleu: 30.10756\tLR: 0.00001211\t\r\n",
      "Steps: 338000\tLoss: 39006.99219\tPPL: 3.73454\tbleu: 30.01059\tLR: 0.00001211\t\r\n",
      "Steps: 339000\tLoss: 38996.47266\tPPL: 3.73322\tbleu: 30.05776\tLR: 0.00001211\t\r\n",
      "Steps: 340000\tLoss: 38991.97266\tPPL: 3.73265\tbleu: 30.15000\tLR: 0.00000847\t\r\n",
      "Steps: 341000\tLoss: 38999.38281\tPPL: 3.73358\tbleu: 30.16355\tLR: 0.00000847\t\r\n",
      "Steps: 342000\tLoss: 38997.77734\tPPL: 3.73338\tbleu: 30.03832\tLR: 0.00000847\t\r\n",
      "Steps: 343000\tLoss: 38996.99219\tPPL: 3.73328\tbleu: 29.97403\tLR: 0.00000847\t\r\n",
      "Steps: 344000\tLoss: 39000.03125\tPPL: 3.73367\tbleu: 30.08525\tLR: 0.00000847\t\r\n",
      "Steps: 345000\tLoss: 38993.23047\tPPL: 3.73281\tbleu: 30.10635\tLR: 0.00000847\t\r\n",
      "Steps: 346000\tLoss: 38997.36719\tPPL: 3.73333\tbleu: 30.18879\tLR: 0.00000593\t\r\n",
      "Steps: 347000\tLoss: 38983.66797\tPPL: 3.73160\tbleu: 30.09061\tLR: 0.00000593\t\r\n",
      "Steps: 348000\tLoss: 38973.27344\tPPL: 3.73029\tbleu: 30.25252\tLR: 0.00000593\t*\r\n",
      "Steps: 349000\tLoss: 38997.27344\tPPL: 3.73332\tbleu: 30.16098\tLR: 0.00000593\t\r\n",
      "Steps: 350000\tLoss: 38990.50000\tPPL: 3.73246\tbleu: 30.06876\tLR: 0.00000593\t\r\n",
      "Steps: 351000\tLoss: 38969.50391\tPPL: 3.72982\tbleu: 30.08651\tLR: 0.00000593\t*\r\n",
      "Steps: 352000\tLoss: 38990.25781\tPPL: 3.73243\tbleu: 30.09612\tLR: 0.00000593\t\r\n",
      "Steps: 353000\tLoss: 38980.91016\tPPL: 3.73126\tbleu: 30.08031\tLR: 0.00000593\t\r\n",
      "Steps: 354000\tLoss: 38989.57812\tPPL: 3.73235\tbleu: 30.04313\tLR: 0.00000593\t\r\n",
      "Steps: 355000\tLoss: 38996.75391\tPPL: 3.73325\tbleu: 30.03178\tLR: 0.00000593\t\r\n",
      "Steps: 356000\tLoss: 38981.08984\tPPL: 3.73128\tbleu: 30.04176\tLR: 0.00000593\t\r\n",
      "Steps: 357000\tLoss: 38993.39844\tPPL: 3.73283\tbleu: 30.19796\tLR: 0.00000415\t\r\n",
      "Steps: 358000\tLoss: 38985.89844\tPPL: 3.73188\tbleu: 30.21084\tLR: 0.00000415\t\r\n",
      "Steps: 359000\tLoss: 38976.90625\tPPL: 3.73075\tbleu: 30.11917\tLR: 0.00000415\t\r\n",
      "Steps: 360000\tLoss: 38968.85156\tPPL: 3.72974\tbleu: 30.08448\tLR: 0.00000415\t*\r\n",
      "Steps: 361000\tLoss: 38975.46484\tPPL: 3.73057\tbleu: 30.03181\tLR: 0.00000415\t\r\n",
      "Steps: 362000\tLoss: 38981.88672\tPPL: 3.73138\tbleu: 30.13214\tLR: 0.00000415\t\r\n",
      "Steps: 363000\tLoss: 38988.39844\tPPL: 3.73220\tbleu: 30.02940\tLR: 0.00000291\t\r\n",
      "Steps: 364000\tLoss: 38973.61719\tPPL: 3.73034\tbleu: 30.05820\tLR: 0.00000291\t\r\n",
      "Steps: 365000\tLoss: 38963.60547\tPPL: 3.72907\tbleu: 30.11077\tLR: 0.00000291\t*\r\n",
      "Steps: 366000\tLoss: 38973.38281\tPPL: 3.73031\tbleu: 30.02222\tLR: 0.00000291\t\r\n",
      "Steps: 367000\tLoss: 38976.57812\tPPL: 3.73071\tbleu: 30.13529\tLR: 0.00000291\t\r\n",
      "Steps: 368000\tLoss: 38968.07031\tPPL: 3.72964\tbleu: 30.05856\tLR: 0.00000291\t\r\n"
     ]
    }
   ],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"$experiment_path/models/${src}${tgt}_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "66WhRE9lIhoD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-13 08:07:47,711 Hello! This is Joey-NMT.\n",
      "2020-02-13 08:08:17,568  dev bleu:  30.29 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
      "2020-02-13 08:08:23,920 test bleu:  44.70 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
     ]
    }
   ],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "starter_notebook.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}