{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Reverse Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## NB\n",
    ">### - The purpose of this Notebook is to build models that translate African languages(target language) *into* English(source language). This will allow us to in future be able to make translations from one African language to the other. If you'd like to translate *from* English, please use [this](https://github.com/masakhane-io/masakhane-mt/blob/master/starter_notebook.ipynb) starter notebook instead.\n",
    "\n",
    ">### - We call this reverse training because normally we build models that make translations from the source language(English) to the target language. But in this case we are doing the reverse; building models that make translations from the target language to the source(English)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "oGRmDELn7Az0",
    "outputId": "807c3318-4a3e-483b-fed0-142c2b9f926a"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'google.colab'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-d5df0069828e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mgoogle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolab\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdrive\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mdrive\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'/content/drive'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'google.colab'"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"sw\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# # This will save it to a folder in our gdrive instead!\n",
    "# !mkdir -p \"/content/drive/My Drive/masakhane/$tgt-$src-$tag\"\n",
    "# os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (target_language, source_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "kBSgJHEw7Nvx",
    "outputId": "3bdb0703-874f-4d53-e5b2-7802a937720b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/masakhane/en-sw-baseline\n"
     ]
    }
   ],
   "source": [
    "# !echo $gdrive_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "gA75Fs9ys8Y9",
    "outputId": "e415db4b-66fe-4785-b80f-6e375809c1bf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting opustools-pkg\n",
      "  Downloading opustools_pkg-0.0.52-py3-none-any.whl (80 kB)\n",
      "\u001b[K     |████████████████████████████████| 80 kB 5.6 MB/s  eta 0:00:01\n",
      "\u001b[?25hInstalling collected packages: opustools-pkg\n",
      "Successfully installed opustools-pkg-0.0.52\n"
     ]
    }
   ],
   "source": [
    "# Install opus-tools\n",
    "! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "xq-tDZVks7ZD",
    "outputId": "c19ee6c0-e652-486c-a1fd-0c5fd265b204"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-sw.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "        ./JW300_latest_xml_en.zip already exists\n",
      "        ./JW300_latest_xml_sw.zip already exists\n",
      "  10 MB https://object.pouta.csc.fi/OPUS-JW300/v1b/xml/en-sw.xml.gz\n",
      "\n",
      "  10 MB Total size\n",
      "./JW300_latest_xml_en-sw.xml.gz ... 100% of 10 MB\n",
      "gzip: JW300_latest_xml_en-sw.xml already exists; do you wish to overwrite (y or n)? ^C\n"
     ]
    }
   ],
   "source": [
    "# Downloading our corpus\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 590
    },
    "id": "n48GDRnP8y2G",
    "outputId": "e27c546b-8b90-43c3-9123-738d3473f12a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-05-10 11:25:43--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en.1’\n",
      "\n",
      "test.en-any.en.1    100%[===================>] 271.28K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2021-05-10 11:25:43 (7.36 MB/s) - ‘test.en-any.en.1’ saved [277791/277791]\n",
      "\n",
      "--2021-05-10 11:25:43--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-sw.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 206336 (202K) [text/plain]\n",
      "Saving to: ‘test.en-sw.en’\n",
      "\n",
      "test.en-sw.en       100%[===================>] 201.50K  --.-KB/s    in 0.02s   \n",
      "\n",
      "2021-05-10 11:25:43 (7.98 MB/s) - ‘test.en-sw.en’ saved [206336/206336]\n",
      "\n",
      "--2021-05-10 11:25:44--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-sw.sw\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 214836 (210K) [text/plain]\n",
      "Saving to: ‘test.en-sw.sw’\n",
      "\n",
      "test.en-sw.sw       100%[===================>] 209.80K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2021-05-10 11:25:44 (15.4 MB/s) - ‘test.en-sw.sw’ saved [214836/214836]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "NqDG-CI28y2L",
    "outputId": "20dc514e-b0d4-4c53-c492-705e42fbbe7c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "id": "3CNdwLBCfSIl",
    "outputId": "fc8f80b7-2f12-4f4b-ca94-3eb0258de6ee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 6478/979526 lines since contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>“ Look !</td>\n",
       "      <td>“ Tazama !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I Am Making All Things New ”</td>\n",
       "      <td>Mimi Ninafanya Vitu Vyote Kuwa Vipya ”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The above is a promise from God that has been ...</td>\n",
       "      <td>Iliyopo juu ni ahadi itokayo kwa Mungu ambayo ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence  \\\n",
       "0                                           “ Look !   \n",
       "1                       I Am Making All Things New ”   \n",
       "2  The above is a promise from God that has been ...   \n",
       "\n",
       "                                     target_sentence  \n",
       "0                                         “ Tazama !  \n",
       "1             Mimi Ninafanya Vitu Vyote Kuwa Vipya ”  \n",
       "2  Iliyopo juu ni ahadi itokayo kwa Mungu ambayo ...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)             \n",
    "with open(target_file) as f:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 191
    },
    "id": "M_2ouEOH1_1q",
    "outputId": "b30fcc33-5a31-46fe-a224-c7d9c4e5bcb5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  import sys\n",
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 278
    },
    "id": "Z_1BwAApEtMk",
    "outputId": "d99a471a-6fb6-4941-950c-7c90f2a0789d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fuzzywuzzy\n",
      "  Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)\n",
      "Installing collected packages: fuzzywuzzy\n",
      "Successfully installed fuzzywuzzy-0.18.0\n",
      "Collecting python-Levenshtein\n",
      "  Downloading python-Levenshtein-0.12.2.tar.gz (50 kB)\n",
      "\u001b[K     |████████████████████████████████| 50 kB 5.3 MB/s  eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from python-Levenshtein) (49.6.0.post20210108)\n",
      "Building wheels for collected packages: python-Levenshtein\n",
      "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.2-cp37-cp37m-linux_x86_64.whl size=178072 sha256=79dd8af974a3293c3d4f3e292e1e6bc3631cff89ddd823545b39199678748f61\n",
      "  Stored in directory: /home/freshiasackey_gmail_com/.cache/pip/wheels/05/5f/ca/7c4367734892581bb5ff896f15027a932c551080b2abd3e00d\n",
      "Successfully built python-Levenshtein\n",
      "Installing collected packages: python-Levenshtein\n",
      "Successfully installed python-Levenshtein-0.12.2\n"
     ]
    }
   ],
   "source": [
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "from os import cpu_count\n",
    "from functools import partial\n",
    "from multiprocessing import Pool\n",
    "\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_pp.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "92EsgTaY3B4H"
   },
   "outputs": [],
   "source": [
    "# start_time = time.time()\n",
    "# ### iterating over pandas dataframe rows is not recomended, let use multi processing to apply the function\n",
    "\n",
    "# with Pool(cpu_count()-1) as pool:\n",
    "#     scores = pool.map(partial(fuzzfilter, candidates=list(en_test_sents), pad=5), df_pp['source_sentence'])\n",
    "# hours, rem = divmod(time.time() - start_time, 3600)\n",
    "# minutes, seconds = divmod(rem, 60)\n",
    "# print(\"done in {}h:{}min:{}seconds\".format(hours, minutes, seconds))\n",
    "\n",
    "# # Filter out \"almost overlapping samples\"\n",
    "# df_pp = df_pp.assign(scores=scores)\n",
    "# df_pp = df_pp[df_pp['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 836
    },
    "id": "hxxBOCA-xXhy",
    "outputId": "e009b507-337c-4c33-e07a-172790f9755f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "Besides , being single gives one the opportunity to do so many great things !\n",
      "A Logical Conclusion\n",
      "I still feel the pain of the loss , but thanking Jehovah for having had a good marriage and for the privilege of serving him with someone who deeply loved him has improved my outlook . ”\n",
      "If we do his will , he blesses and supports us in this life and offers us the certain hope of an even better life to come .\n",
      "We are determined to continue providing a rich supply of enlightening and appealing information ​ — both in print and online — ​ to benefit our many readers who respect the Bible and who want to know what it really teaches .\n",
      "Well , in the past decade , the number of active Witnesses increased from slightly fewer than 3,800,000 to almost 6,000,000 .\n",
      "Actually , according to the Bible , there is no spiritual part of a person that survives the death of the body .\n",
      "The enemy may aim a blow by attacking Bible truths that are fundamental to your faith .\n",
      "When Saul returned to Damascus , “ the Jews took counsel together to do away with him . ”\n",
      "To settle the issue of Godship , Elijah now proposes a contest .\n",
      "\n",
      "==> train.sw <==\n",
      "Isitoshe , kwa kuwa sijaanzisha urafiki kama huo nina nafasi ya kufanya mambo mengi ya pekee !\n",
      "Mkataa Unaopatana na Akili\n",
      "Bado nina huzuni ya kufiwa , lakini mtazamo wangu umebadilika kwa sababu ya kumshukuru Yehova kwa kuwa nilikuwa na ndoa yenye furaha na pendeleo la kumtumikia pamoja na mtu aliyempenda sana . ”\n",
      "Tukifanya mapenzi yake , yeye hutubariki na hututegemeza katika uhai huu na hututolea lile tumaini hakika la uhai ulio mzuri hata zaidi utakaokuja .\n",
      "Tumeazimia kuendelea kutoa habari nyingi zenye kuelimisha na zinazovutia — zinazochapishwa na pia zinazopatikana kwenye Intaneti — ili kuwafaidi wasomaji wetu wengi wanaoiheshimu Biblia na wanaotaka kujua inafundisha nini hasa .\n",
      "Katika miaka kumi iliyopita , idadi ya Mashahidi watendaji iliongezeka kutoka 3,800,000 hivi hadi karibu 6,000,000 .\n",
      "Kwa kweli , kulingana na Biblia , hakuna sehemu yoyote ya kiroho ya mtu ambayo huendelea kuwa hai baada ya kufa kwa mwili .\n",
      "Huenda adui akaamua kushambulia kweli za Biblia ambazo ni muhimu kwa imani yako .\n",
      "Sauli aliporudi Damasko , “ Wayahudi walikata shauri pamoja kumwangamiza . ”\n",
      "Ili kusuluhisha hilo suala la Uungu , Eliya apendekeza shindano .\n",
      "==> dev.en <==\n",
      "There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "Lots of hugs and kisses .\n",
      "When Harald was arrested , his wife , Elsa , was still breast - feeding their ten - month - old baby girl .\n",
      "Still , from his painful experience , he learned some valuable lessons .\n",
      "How did Pharaoh show haughtiness , and with what result ?\n",
      "What seems to trigger such outbursts ?\n",
      "In a research project , Poole heard a sound that was different from the normal elephant calls .\n",
      "Misery will give way to happiness when God blesses you as he did Job .\n",
      "\n",
      "==> dev.sw <==\n",
      "Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "Kukumbatia kwingi na busu nyingi .\n",
      "Wakati Harald alipokamatwa , mke wake , Elsa , alikuwa bado anamnyonyesha binti yao mwenye umri wa miezi kumi .\n",
      "Ingawa hakupata utajiri , kisa chake chenye kuhuzunisha kilimfunza mambo muhimu .\n",
      "Farao alionyeshaje majivuno , na matokeo yalikuwa nini ?\n",
      "Ni nini huchochea milipuko hiyo ya hasira ?\n",
      "Poole alipokuwa akifanya uchunguzi fulani , alisikia mlio uliokuwa tofauti na milio ya kawaida ya tembo .\n",
      "Furaha itachukua mahali pa huzuni Mungu anapokubariki kama alivyombariki Ayubu .\n"
     ]
    }
   ],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "iBRMm4kMxZ8L",
    "outputId": "97a36759-3f50-4498-f54e-9d36cb35c9ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'joeynmt'...\n",
      "remote: Enumerating objects: 3089, done.\u001b[K\n",
      "remote: Counting objects: 100% (138/138), done.\u001b[K\n",
      "remote: Compressing objects: 100% (65/65), done.\u001b[K\n",
      "remote: Total 3089 (delta 78), reused 118 (delta 73), pack-reused 2951\u001b[K\n",
      "Receiving objects: 100% (3089/3089), 8.07 MiB | 20.07 MiB/s, done.\n",
      "Resolving deltas: 100% (2107/2107), done.\n",
      "Processing /home/freshiasackey_gmail_com/joeynmt\n",
      "Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (0.18.2)\n",
      "Requirement already satisfied: pillow in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (7.2.0)\n",
      "Requirement already satisfied: numpy==1.20.1 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (1.20.1)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (49.6.0.post20210108)\n",
      "Requirement already satisfied: torch==1.8.0 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (1.8.0)\n",
      "Requirement already satisfied: tensorboard>=1.15 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (2.5.0)\n",
      "Requirement already satisfied: torchtext==0.9.0 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (0.9.0)\n",
      "Requirement already satisfied: sacrebleu>=1.3.6 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (1.5.1)\n",
      "Requirement already satisfied: subword-nmt in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (0.3.7)\n",
      "Requirement already satisfied: matplotlib in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (3.4.1)\n",
      "Requirement already satisfied: seaborn in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (0.11.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (5.4.1)\n",
      "Requirement already satisfied: pylint in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (2.8.2)\n",
      "Requirement already satisfied: six==1.12 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (1.12.0)\n",
      "Requirement already satisfied: wrapt==1.11.1 in /opt/conda/lib/python3.7/site-packages (from joeynmt==1.3) (1.11.1)\n",
      "Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.8.0->joeynmt==1.3) (3.7.4.3)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from torchtext==0.9.0->joeynmt==1.3) (4.60.0)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from torchtext==0.9.0->joeynmt==1.3) (2.25.1)\n",
      "Requirement already satisfied: portalocker==2.0.0 in /opt/conda/lib/python3.7/site-packages (from sacrebleu>=1.3.6->joeynmt==1.3) (2.0.0)\n",
      "Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (3.15.8)\n",
      "Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (0.36.2)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (0.6.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (0.4.3)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (1.8.0)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (1.28.0)\n",
      "Requirement already satisfied: absl-py>=0.4 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (0.12.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (3.3.4)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (1.0.1)\n",
      "Requirement already satisfied: grpcio>=1.24.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard>=1.15->joeynmt==1.3) (1.37.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==1.3) (4.7.2)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==1.3) (4.2.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==1.3) (0.2.7)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt==1.3) (1.3.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard>=1.15->joeynmt==1.3) (3.10.1)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==1.3) (0.4.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->torchtext==0.9.0->joeynmt==1.3) (2020.12.5)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->torchtext==0.9.0->joeynmt==1.3) (4.0.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->torchtext==0.9.0->joeynmt==1.3) (1.26.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->torchtext==0.9.0->joeynmt==1.3) (2.10)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt==1.3) (3.0.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard>=1.15->joeynmt==1.3) (3.4.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib->joeynmt==1.3) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.7/site-packages (from matplotlib->joeynmt==1.3) (2.8.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->joeynmt==1.3) (1.3.1)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->joeynmt==1.3) (2.4.7)\n",
      "Requirement already satisfied: mccabe<0.7,>=0.6 in /opt/conda/lib/python3.7/site-packages (from pylint->joeynmt==1.3) (0.6.1)\n",
      "Requirement already satisfied: toml>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from pylint->joeynmt==1.3) (0.10.2)\n",
      "Requirement already satisfied: isort<6,>=4.2.5 in /opt/conda/lib/python3.7/site-packages (from pylint->joeynmt==1.3) (5.8.0)\n",
      "Requirement already satisfied: astroid<2.7,>=2.5.6 in /opt/conda/lib/python3.7/site-packages (from pylint->joeynmt==1.3) (2.5.6)\n",
      "Requirement already satisfied: lazy-object-proxy>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from astroid<2.7,>=2.5.6->pylint->joeynmt==1.3) (1.6.0)\n",
      "Requirement already satisfied: typed-ast<1.5,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from astroid<2.7,>=2.5.6->pylint->joeynmt==1.3) (1.4.3)\n",
      "Requirement already satisfied: pandas>=0.23 in /opt/conda/lib/python3.7/site-packages (from seaborn->joeynmt==1.3) (1.2.4)\n",
      "Requirement already satisfied: scipy>=1.0 in /opt/conda/lib/python3.7/site-packages (from seaborn->joeynmt==1.3) (1.6.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.23->seaborn->joeynmt==1.3) (2021.1)\n",
      "Building wheels for collected packages: joeynmt\n",
      "  Building wheel for joeynmt (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for joeynmt: filename=joeynmt-1.3-py3-none-any.whl size=84842 sha256=ef07bebe91da73788bfd625d9ea9f6c2d4e41abd48a82cee881ca4aae778845b\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-i4o_on4y/wheels/c6/e5/f6/9ca9cce28ddbc86321064c68c12df09503f033db723fa4548d\n",
      "Successfully built joeynmt\n",
      "Installing collected packages: joeynmt\n",
      "  Attempting uninstall: joeynmt\n",
      "    Found existing installation: joeynmt 1.3\n",
      "    Uninstalling joeynmt-1.3:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Successfully uninstalled joeynmt-1.3\n",
      "Successfully installed joeynmt-1.3\n",
      "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
      "Collecting torch==1.8.0+cu101\n",
      "  Downloading https://download.pytorch.org/whl/cu101/torch-1.8.0%2Bcu101-cp37-cp37m-linux_x86_64.whl (763.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 763.5 MB 9.8 kB/s eta 0:00:012     |██████████████████▏             | 432.3 MB 7.8 MB/s eta 0:00:43     |███████████████████▊            | 469.5 MB 2.3 MB/s eta 0:02:09\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.8.0+cu101) (3.7.4.3)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torch==1.8.0+cu101) (1.20.1)\n",
      "Installing collected packages: torch\n",
      "  Attempting uninstall: torch\n",
      "    Found existing installation: torch 1.8.0\n",
      "    Uninstalling torch-1.8.0:\n",
      "      Successfully uninstalled torch-1.8.0\n",
      "Successfully installed torch-1.8.0+cu101\n"
     ]
    }
   ],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip3 install .\n",
    "# Install Pytorch with GPU support v1.8.0\n",
    "! pip install torch==1.8.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 416
    },
    "id": "H-TyjtmXB1mL",
    "outputId": "9867ad89-e5ec-4007-b952-d73a158ff5cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.sw     test.en-any.en.1  train.bpe.sw\n",
      "dev.bpe.en\tdev.sw\t     test.en\t     test.sw\t       train.en\n",
      "dev.bpe.sw\ttest.bpe.en  test.en-any.en  train.bpe.en      train.sw\n",
      "cp: target '' is not a directory\n",
      "cp: target '' is not a directory\n",
      "cp: target '' is not a directory\n",
      "cp: cannot create regular file '': No such file or directory\n",
      "ls: cannot access '': No such file or directory\n",
      "BPE Swahili Sentences\n",
      "N@@ g@@ ao kubwa ya imani ( T@@ azama f@@ ungu la 12 hadi 14 )\n",
      "K@@ of@@ ia ya chu@@ ma ya wo@@ ko@@ vu ( T@@ azama f@@ ungu la 15 hadi 18 )\n",
      "N@@ im@@ et@@ ambua kwamba watu hu@@ it@@ ikia vizuri wanapo@@ ona uki@@ zungum@@ zia habari za Biblia kwa sha@@ uku na unapo@@ fanya yote una@@ yo@@ weza kuwasaidia . ”\n",
      "Up@@ anga wa roho ( T@@ azama f@@ ungu la 19 na 20 )\n",
      "Kwa msaada wa Yehova tunaweza kus@@ im@@ ama im@@ ara na kum@@ p@@ inga !\n",
      "Combined BPE Vocab\n",
      "с@@\n",
      "ḥ\n",
      "т@@\n",
      "и\n",
      "х@@\n",
      "м\n",
      "і@@\n",
      "д@@\n",
      "і\n",
      "➊\n"
     ]
    }
   ],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\",target_language + source_language ) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$tgt$src/train.bpe.$src joeynmt/data/$tgt$src/train.bpe.$tgt --output_path joeynmt/data/$tgt$src/vocab.txt\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Swahili Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 joeynmt/data/$tgt$src/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "id": "IlMitUHR8Qy-",
    "outputId": "1656a274-cddb-47a2-bf71-6f64c5ef8a24"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.sw     test.en-any.en.1  train.bpe.sw\n",
      "dev.bpe.en\tdev.sw\t     test.en\t     test.sw\t       train.en\n",
      "dev.bpe.sw\ttest.bpe.en  test.en-any.en  train.bpe.en      train.sw\n"
     ]
    }
   ],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "h8TMgv1p3L1z"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (target_language, source_language)\n",
    "# gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{target_language}{source_language}_reverse_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{target_language}\"\n",
    "    trg: \"{source_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 35                  # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_reverse_transformer\"\n",
    "    overwrite: True              # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\"\n",
    "with open(\"joeynmt/configs/transformer_reverse_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oEzoJtV2MIpt"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "WzbNYNdjLgNb",
    "outputId": "cc18bde5-6226-4769-de84-5e5d27a6c5a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 12:05:13,638 - INFO - root - Hello! This is Joey-NMT (version 1.3).\n",
      "2021-05-10 12:05:13,639 - INFO - joeynmt.data - Loading training data...\n",
      "2021-05-10 12:05:33,860 - INFO - joeynmt.data - Building vocabulary...\n",
      "2021-05-10 12:05:34,175 - INFO - joeynmt.data - Loading dev data...\n",
      "2021-05-10 12:05:34,236 - INFO - joeynmt.data - Loading test data...\n",
      "2021-05-10 12:05:34,322 - INFO - joeynmt.data - Data loaded.\n",
      "2021-05-10 12:05:34,322 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2021-05-10 12:05:34,569 - INFO - joeynmt.model - Enc-dec model built.\n",
      "2021-05-10 12:05:34,577 - INFO - joeynmt.training - Total params: 12219392\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.name                           : {target_language}{source_language}_reverse_transformer\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.src                       : sw\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.trg                       : en\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.train                     : data/swen/train.bpe\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.dev                       : data/swen/dev.bpe\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.test                      : data/swen/test.bpe\n",
      "2021-05-10 12:05:34,580 - INFO - joeynmt.helpers - cfg.data.level                     : bpe\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.data.lowercase                 : False\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.data.max_sent_length           : 100\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.data.src_vocab                 : data/swen/vocab.txt\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.data.trg_vocab                 : data/swen/vocab.txt\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.testing.beam_size              : 5\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.testing.alpha                  : 1.0\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.random_seed           : 42\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.optimizer             : adam\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.normalization         : tokens\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.scheduling            : plateau\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.patience              : 5\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.learning_rate_factor  : 0.5\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.learning_rate_warmup  : 1000\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.decrease_factor       : 0.7\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.loss                  : crossentropy\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.learning_rate         : 0.0003\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.learning_rate_min     : 1e-08\n",
      "2021-05-10 12:05:34,581 - INFO - joeynmt.helpers - cfg.training.weight_decay          : 0.0\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.label_smoothing       : 0.1\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.batch_size            : 4096\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.batch_type            : token\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.eval_batch_size       : 3600\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.eval_batch_type       : token\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.batch_multiplier      : 1\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.early_stopping_metric : ppl\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.epochs                : 35\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.validation_freq       : 1000\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.logging_freq          : 100\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.eval_metric           : bleu\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.model_dir             : models/swen_reverse_transformer\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.overwrite             : True\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.shuffle               : True\n",
      "2021-05-10 12:05:34,582 - INFO - joeynmt.helpers - cfg.training.use_cuda              : True\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.training.max_output_length     : 100\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.training.keep_last_ckpts       : 3\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.initializer              : xavier\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.bias_initializer         : zeros\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.init_gain                : 1.0\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.embed_initializer        : xavier\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.embed_init_gain          : 1.0\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.tied_embeddings          : True\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.tied_softmax             : True\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.type             : transformer\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.num_layers       : 6\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.num_heads        : 4\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.scale : True\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2021-05-10 12:05:34,583 - INFO - joeynmt.helpers - cfg.model.encoder.hidden_size      : 256\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.encoder.ff_size          : 1024\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.encoder.dropout          : 0.3\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.type             : transformer\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.num_layers       : 6\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.num_heads        : 4\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.scale : True\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.hidden_size      : 256\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.ff_size          : 1024\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - cfg.model.decoder.dropout          : 0.3\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - Data set sizes: \n",
      "\ttrain 873221,\n",
      "\tvalid 1000,\n",
      "\ttest 2721\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - First training example:\n",
      "\t[SRC] Is@@ ito@@ she , kwa kuwa si@@ j@@ a@@ anzisha ur@@ afiki kama huo n@@ ina nafasi ya kufanya mambo mengi ya pekee !\n",
      "\t[TRG] B@@ es@@ id@@ es , being sing@@ le gives one the opportun@@ ity to do so many great things !\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) the (7) na (8) ya (9) to\n",
      "2021-05-10 12:05:34,584 - INFO - joeynmt.helpers - First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) the (7) na (8) ya (9) to\n",
      "2021-05-10 12:05:34,585 - INFO - joeynmt.helpers - Number of Src words (types): 4528\n",
      "2021-05-10 12:05:34,585 - INFO - joeynmt.helpers - Number of Trg words (types): 4528\n",
      "2021-05-10 12:05:34,585 - INFO - joeynmt.training - Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4528),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4528))\n",
      "2021-05-10 12:05:34,589 - INFO - joeynmt.training - Train stats:\n",
      "\tdevice: cpu\n",
      "\tn_gpu: 0\n",
      "\t16-bits training: False\n",
      "\tgradient accumulation: 1\n",
      "\tbatch size per device: 4096\n",
      "\ttotal batch size (w. parallel & accumulation): 4096\n",
      "2021-05-10 12:05:34,589 - INFO - joeynmt.training - EPOCH 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 12:07:14,714 - INFO - joeynmt.training - Epoch   1, Step:      100, Batch Loss:     5.824012, Tokens per Sec:     2461, Lr: 0.000300\n",
      "2021-05-10 12:08:53,289 - INFO - joeynmt.training - Epoch   1, Step:      200, Batch Loss:     5.505569, Tokens per Sec:     2571, Lr: 0.000300\n",
      "2021-05-10 12:10:31,850 - INFO - joeynmt.training - Epoch   1, Step:      300, Batch Loss:     5.457898, Tokens per Sec:     2620, Lr: 0.000300\n",
      "2021-05-10 12:12:07,941 - INFO - joeynmt.training - Epoch   1, Step:      400, Batch Loss:     5.160206, Tokens per Sec:     2542, Lr: 0.000300\n",
      "2021-05-10 12:13:44,283 - INFO - joeynmt.training - Epoch   1, Step:      500, Batch Loss:     5.064879, Tokens per Sec:     2604, Lr: 0.000300\n",
      "2021-05-10 12:15:19,514 - INFO - joeynmt.training - Epoch   1, Step:      600, Batch Loss:     5.153046, Tokens per Sec:     2587, Lr: 0.000300\n",
      "2021-05-10 12:16:57,236 - INFO - joeynmt.training - Epoch   1, Step:      700, Batch Loss:     4.622757, Tokens per Sec:     2603, Lr: 0.000300\n",
      "2021-05-10 12:18:32,836 - INFO - joeynmt.training - Epoch   1, Step:      800, Batch Loss:     4.622123, Tokens per Sec:     2537, Lr: 0.000300\n",
      "2021-05-10 12:21:44,480 - INFO - joeynmt.training - Epoch   1, Step:     1000, Batch Loss:     4.562954, Tokens per Sec:     2630, Lr: 0.000300\n",
      "2021-05-10 12:25:39,140 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 12:25:39,141 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 12:25:39,141 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 12:25:39,392 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 12:25:39,392 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tHypothesis: The year of the year of the year .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tHypothesis: The Bible is the Bible of the Bible of the Bible of the Bible .\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 12:25:39,697 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - \tHypothesis: The Bible of the Bible , the Bible of the Bible , the Bible of the Bible , the Bible of the Bible of the Bible , the Bible of the Bible .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - \tHypothesis: The Bible was a time of the Bible .\n",
      "2021-05-10 12:25:39,698 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     1000: bleu:   1.06, loss: 129890.1797, ppl:  92.6316, duration: 235.2171s\n",
      "2021-05-10 12:27:15,979 - INFO - joeynmt.training - Epoch   1, Step:     1100, Batch Loss:     4.374075, Tokens per Sec:     2656, Lr: 0.000300\n",
      "2021-05-10 12:28:52,823 - INFO - joeynmt.training - Epoch   1, Step:     1200, Batch Loss:     4.553164, Tokens per Sec:     2646, Lr: 0.000300\n",
      "2021-05-10 12:30:20,386 - INFO - joeynmt.training - Epoch   1, Step:     1300, Batch Loss:     4.502044, Tokens per Sec:     2833, Lr: 0.000300\n",
      "2021-05-10 12:31:48,780 - INFO - joeynmt.training - Epoch   1, Step:     1400, Batch Loss:     4.345503, Tokens per Sec:     2861, Lr: 0.000300\n",
      "2021-05-10 12:33:17,365 - INFO - joeynmt.training - Epoch   1, Step:     1500, Batch Loss:     4.163601, Tokens per Sec:     2830, Lr: 0.000300\n",
      "2021-05-10 12:34:44,243 - INFO - joeynmt.training - Epoch   1, Step:     1600, Batch Loss:     4.173061, Tokens per Sec:     2821, Lr: 0.000300\n",
      "2021-05-10 12:36:10,735 - INFO - joeynmt.training - Epoch   1, Step:     1700, Batch Loss:     4.261605, Tokens per Sec:     2799, Lr: 0.000300\n",
      "2021-05-10 12:37:38,804 - INFO - joeynmt.training - Epoch   1, Step:     1800, Batch Loss:     4.135508, Tokens per Sec:     2841, Lr: 0.000300\n",
      "2021-05-10 12:39:06,062 - INFO - joeynmt.training - Epoch   1, Step:     1900, Batch Loss:     4.002204, Tokens per Sec:     2853, Lr: 0.000300\n",
      "2021-05-10 12:40:33,729 - INFO - joeynmt.training - Epoch   1, Step:     2000, Batch Loss:     4.081915, Tokens per Sec:     2874, Lr: 0.000300\n",
      "2021-05-10 12:44:26,202 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 12:44:26,202 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 12:44:26,202 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 12:44:26,490 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 12:44:26,491 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 12:44:26,689 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 12:44:26,689 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tHypothesis: In 196th century of the United States .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tHypothesis: The Bible will be a Bible that we have been a Bible will be a Bible and the Bible and the Bible and his name .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - \tHypothesis: The Shal of the Shaocer , the Cran , the Cran , the Cranch of the Cranh , the Cren’s , the sleep of the world .\n",
      "2021-05-10 12:44:26,690 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 12:44:26,691 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 12:44:26,691 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 12:44:26,691 - INFO - joeynmt.training - \tHypothesis: The source of the sleep and the sleep .\n",
      "2021-05-10 12:44:26,691 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     2000: bleu:   1.91, loss: 110982.6562, ppl:  47.9145, duration: 232.9610s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 12:45:54,109 - INFO - joeynmt.training - Epoch   1, Step:     2100, Batch Loss:     4.114491, Tokens per Sec:     2871, Lr: 0.000300\n",
      "2021-05-10 12:47:21,622 - INFO - joeynmt.training - Epoch   1, Step:     2200, Batch Loss:     3.484627, Tokens per Sec:     2874, Lr: 0.000300\n",
      "2021-05-10 12:50:18,393 - INFO - joeynmt.training - Epoch   1, Step:     2400, Batch Loss:     3.704122, Tokens per Sec:     2867, Lr: 0.000300\n",
      "2021-05-10 12:51:45,707 - INFO - joeynmt.training - Epoch   1, Step:     2500, Batch Loss:     3.647293, Tokens per Sec:     2794, Lr: 0.000300\n",
      "2021-05-10 12:53:14,079 - INFO - joeynmt.training - Epoch   1, Step:     2600, Batch Loss:     3.703819, Tokens per Sec:     2880, Lr: 0.000300\n",
      "2021-05-10 12:54:38,979 - INFO - joeynmt.training - Epoch   1, Step:     2700, Batch Loss:     3.674768, Tokens per Sec:     2973, Lr: 0.000300\n",
      "2021-05-10 12:56:04,201 - INFO - joeynmt.training - Epoch   1, Step:     2800, Batch Loss:     3.530214, Tokens per Sec:     2920, Lr: 0.000300\n",
      "2021-05-10 12:57:32,161 - INFO - joeynmt.training - Epoch   1, Step:     2900, Batch Loss:     3.656303, Tokens per Sec:     2791, Lr: 0.000300\n",
      "2021-05-10 12:59:01,675 - INFO - joeynmt.training - Epoch   1, Step:     3000, Batch Loss:     3.622418, Tokens per Sec:     2793, Lr: 0.000300\n",
      "2021-05-10 13:02:05,468 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 13:02:05,468 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 13:02:05,469 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 13:02:05,752 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 13:02:05,753 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 13:02:05,959 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 13:02:05,959 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 13:02:05,959 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 13:02:05,959 - INFO - joeynmt.training - \tHypothesis: In 1990 years of the year were in the United States .\n",
      "2021-05-10 13:02:05,959 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tHypothesis: The Bible will be a person that he was a Bible and the Bible and his promises .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tHypothesis: The Church in the United States , the Church is a , as a Church , the Church , the South of the Church , the most of their figergy , they were not to be a source of their own .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - \tHypothesis: The pain of the people and the people .\n",
      "2021-05-10 13:02:05,960 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     3000: bleu:   4.24, loss: 98905.2969, ppl:  31.4482, duration: 184.2848s\n",
      "2021-05-10 13:03:34,761 - INFO - joeynmt.training - Epoch   1, Step:     3100, Batch Loss:     3.201812, Tokens per Sec:     2843, Lr: 0.000300\n",
      "2021-05-10 13:05:03,222 - INFO - joeynmt.training - Epoch   1, Step:     3200, Batch Loss:     3.633333, Tokens per Sec:     2815, Lr: 0.000300\n",
      "2021-05-10 13:06:31,416 - INFO - joeynmt.training - Epoch   1, Step:     3300, Batch Loss:     3.959035, Tokens per Sec:     2791, Lr: 0.000300\n",
      "2021-05-10 13:08:00,386 - INFO - joeynmt.training - Epoch   1, Step:     3400, Batch Loss:     3.108100, Tokens per Sec:     2865, Lr: 0.000300\n",
      "2021-05-10 13:09:29,731 - INFO - joeynmt.training - Epoch   1, Step:     3500, Batch Loss:     3.613675, Tokens per Sec:     2847, Lr: 0.000300\n",
      "2021-05-10 13:10:58,629 - INFO - joeynmt.training - Epoch   1, Step:     3600, Batch Loss:     3.363529, Tokens per Sec:     2867, Lr: 0.000300\n",
      "2021-05-10 13:12:27,535 - INFO - joeynmt.training - Epoch   1, Step:     3700, Batch Loss:     3.335052, Tokens per Sec:     2833, Lr: 0.000300\n",
      "2021-05-10 13:15:22,756 - INFO - joeynmt.training - Epoch   1, Step:     3900, Batch Loss:     3.351516, Tokens per Sec:     2842, Lr: 0.000300\n",
      "2021-05-10 13:16:51,232 - INFO - joeynmt.training - Epoch   1, Step:     4000, Batch Loss:     3.097704, Tokens per Sec:     2866, Lr: 0.000300\n",
      "2021-05-10 13:19:04,549 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 13:19:04,549 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 13:19:04,549 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 13:19:04,817 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 13:19:04,817 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 13:19:05,031 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tHypothesis: In 1997 the two two two two were in the city .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tHypothesis: We will be able to know what the Bible is to be a good news and his own own .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tHypothesis: The city of the city in the city of South , which is a city of the Europe , as the Europe of the Europe , the city of the swwwwd , and they were to take their own own .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 13:19:05,032 - INFO - joeynmt.training - \tHypothesis: The refuge of the people and have been reached .\n",
      "2021-05-10 13:19:05,033 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     4000: bleu:   7.61, loss: 90962.1953, ppl:  23.8409, duration: 133.8001s\n",
      "2021-05-10 13:20:34,939 - INFO - joeynmt.training - Epoch   1, Step:     4100, Batch Loss:     3.672779, Tokens per Sec:     2852, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 13:22:02,438 - INFO - joeynmt.training - Epoch   1, Step:     4200, Batch Loss:     2.996407, Tokens per Sec:     2813, Lr: 0.000300\n",
      "2021-05-10 13:23:29,382 - INFO - joeynmt.training - Epoch   1, Step:     4300, Batch Loss:     3.099476, Tokens per Sec:     2794, Lr: 0.000300\n",
      "2021-05-10 13:26:24,939 - INFO - joeynmt.training - Epoch   1, Step:     4500, Batch Loss:     3.238181, Tokens per Sec:     2824, Lr: 0.000300\n",
      "2021-05-10 13:27:52,748 - INFO - joeynmt.training - Epoch   1, Step:     4600, Batch Loss:     3.237501, Tokens per Sec:     2835, Lr: 0.000300\n",
      "2021-05-10 13:29:20,644 - INFO - joeynmt.training - Epoch   1, Step:     4700, Batch Loss:     3.250057, Tokens per Sec:     2844, Lr: 0.000300\n",
      "2021-05-10 13:30:48,449 - INFO - joeynmt.training - Epoch   1, Step:     4800, Batch Loss:     3.238878, Tokens per Sec:     2818, Lr: 0.000300\n",
      "2021-05-10 13:32:16,912 - INFO - joeynmt.training - Epoch   1, Step:     4900, Batch Loss:     2.947474, Tokens per Sec:     2826, Lr: 0.000300\n",
      "2021-05-10 13:33:46,017 - INFO - joeynmt.training - Epoch   1, Step:     5000, Batch Loss:     3.065637, Tokens per Sec:     2805, Lr: 0.000300\n",
      "2021-05-10 13:36:16,599 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 13:36:16,599 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 13:36:16,599 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 13:36:16,880 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 13:36:16,880 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tHypothesis: In 1997 the year of the age of the 30,000 girls were found in the Britain .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tHypothesis: It is not to know what the Bible is to know what the purpose of his purpose and his heart .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 13:36:17,098 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - \tHypothesis: The arrital of the city of Syyran , which is the city of Europe of Europe , as the Europe of the Great , the art of the sleaves , and they were able to be reached their own .\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - \tHypothesis: The most of the most seven and many .\n",
      "2021-05-10 13:36:17,099 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     5000: bleu:   9.73, loss: 84097.4766, ppl:  18.7663, duration: 151.0815s\n",
      "2021-05-10 13:37:46,166 - INFO - joeynmt.training - Epoch   1, Step:     5100, Batch Loss:     3.480074, Tokens per Sec:     2819, Lr: 0.000300\n",
      "2021-05-10 13:39:13,693 - INFO - joeynmt.training - Epoch   1, Step:     5200, Batch Loss:     3.152671, Tokens per Sec:     2823, Lr: 0.000300\n",
      "2021-05-10 13:40:41,526 - INFO - joeynmt.training - Epoch   1, Step:     5300, Batch Loss:     2.737840, Tokens per Sec:     2798, Lr: 0.000300\n",
      "2021-05-10 13:42:10,236 - INFO - joeynmt.training - Epoch   1, Step:     5400, Batch Loss:     3.052401, Tokens per Sec:     2857, Lr: 0.000300\n",
      "2021-05-10 13:43:39,203 - INFO - joeynmt.training - Epoch   1, Step:     5500, Batch Loss:     2.959814, Tokens per Sec:     2858, Lr: 0.000300\n",
      "2021-05-10 13:45:07,252 - INFO - joeynmt.training - Epoch   1, Step:     5600, Batch Loss:     3.237727, Tokens per Sec:     2867, Lr: 0.000300\n",
      "2021-05-10 13:46:35,422 - INFO - joeynmt.training - Epoch   1, Step:     5700, Batch Loss:     2.970789, Tokens per Sec:     2898, Lr: 0.000300\n",
      "2021-05-10 13:48:02,824 - INFO - joeynmt.training - Epoch   1, Step:     5800, Batch Loss:     3.036379, Tokens per Sec:     2788, Lr: 0.000300\n",
      "2021-05-10 13:49:32,111 - INFO - joeynmt.training - Epoch   1, Step:     5900, Batch Loss:     3.152328, Tokens per Sec:     2867, Lr: 0.000300\n",
      "2021-05-10 13:51:00,682 - INFO - joeynmt.training - Epoch   1, Step:     6000, Batch Loss:     2.889906, Tokens per Sec:     2837, Lr: 0.000300\n",
      "2021-05-10 13:54:30,791 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 13:54:30,791 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 13:54:30,791 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 13:54:31,068 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 13:54:31,068 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 13:54:31,280 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tHypothesis: In 1997 a year - year girls were returned in Britain .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tHypothesis: It is not to know what the Bible does not give the purpose of his own and his own .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tHypothesis: The city of Sydddddddddddddddy , which is the Europe of Europe of the Europe of the Messiah , the rejected to their own putes .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 13:54:31,281 - INFO - joeynmt.training - \tHypothesis: The most of the seven and often .\n",
      "2021-05-10 13:54:31,282 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     6000: bleu:  11.24, loss: 79250.9609, ppl:  15.8487, duration: 210.5988s\n",
      "2021-05-10 13:55:59,226 - INFO - joeynmt.training - Epoch   1, Step:     6100, Batch Loss:     2.726814, Tokens per Sec:     2877, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 13:57:27,905 - INFO - joeynmt.training - Epoch   1, Step:     6200, Batch Loss:     2.735085, Tokens per Sec:     2882, Lr: 0.000300\n",
      "2021-05-10 13:58:55,655 - INFO - joeynmt.training - Epoch   1, Step:     6300, Batch Loss:     2.721830, Tokens per Sec:     2781, Lr: 0.000300\n",
      "2021-05-10 14:00:25,340 - INFO - joeynmt.training - Epoch   1, Step:     6400, Batch Loss:     2.639565, Tokens per Sec:     2802, Lr: 0.000300\n",
      "2021-05-10 14:01:53,724 - INFO - joeynmt.training - Epoch   1, Step:     6500, Batch Loss:     3.123513, Tokens per Sec:     2798, Lr: 0.000300\n",
      "2021-05-10 14:03:22,714 - INFO - joeynmt.training - Epoch   1, Step:     6600, Batch Loss:     2.939059, Tokens per Sec:     2823, Lr: 0.000300\n",
      "2021-05-10 14:04:51,919 - INFO - joeynmt.training - Epoch   1, Step:     6700, Batch Loss:     2.742112, Tokens per Sec:     2842, Lr: 0.000300\n",
      "2021-05-10 14:06:21,332 - INFO - joeynmt.training - Epoch   1, Step:     6800, Batch Loss:     3.273561, Tokens per Sec:     2886, Lr: 0.000300\n",
      "2021-05-10 14:07:50,202 - INFO - joeynmt.training - Epoch   1, Step:     6900, Batch Loss:     2.880594, Tokens per Sec:     2897, Lr: 0.000300\n",
      "2021-05-10 14:09:18,899 - INFO - joeynmt.training - Epoch   1, Step:     7000, Batch Loss:     3.233414, Tokens per Sec:     2860, Lr: 0.000300\n",
      "2021-05-10 14:11:39,966 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 14:11:39,966 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 14:11:39,966 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 14:11:40,241 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 14:11:40,241 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 14:11:40,455 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 14:11:40,455 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 14:11:40,455 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 14:11:40,455 - INFO - joeynmt.training - \tHypothesis: In 1997 a 90,000 girls were in Britain .\n",
      "2021-05-10 14:11:40,455 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tHypothesis: It is not to know what the Bible says the right of his own purpose and his heart .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tHypothesis: The city of Syllerg , which is a part of Europe of Europe , as the Europe of the Europe of the Europe , who returned to the sleep of their paper .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - \tHypothesis: The most of the most and many of the seven .\n",
      "2021-05-10 14:11:40,456 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     7000: bleu:  14.56, loss: 75467.2188, ppl:  13.8900, duration: 141.5573s\n",
      "2021-05-10 14:13:08,847 - INFO - joeynmt.training - Epoch   1, Step:     7100, Batch Loss:     2.824364, Tokens per Sec:     2834, Lr: 0.000300\n",
      "2021-05-10 14:14:38,142 - INFO - joeynmt.training - Epoch   1, Step:     7200, Batch Loss:     2.770475, Tokens per Sec:     2837, Lr: 0.000300\n",
      "2021-05-10 14:16:03,753 - INFO - joeynmt.training - Epoch   1, Step:     7300, Batch Loss:     2.589112, Tokens per Sec:     2922, Lr: 0.000300\n",
      "2021-05-10 14:17:28,026 - INFO - joeynmt.training - Epoch   1, Step:     7400, Batch Loss:     3.010032, Tokens per Sec:     3030, Lr: 0.000300\n",
      "2021-05-10 14:18:52,780 - INFO - joeynmt.training - Epoch   1, Step:     7500, Batch Loss:     2.712847, Tokens per Sec:     2955, Lr: 0.000300\n",
      "2021-05-10 14:20:21,534 - INFO - joeynmt.training - Epoch   1, Step:     7600, Batch Loss:     2.793720, Tokens per Sec:     2886, Lr: 0.000300\n",
      "2021-05-10 14:21:49,718 - INFO - joeynmt.training - Epoch   1, Step:     7700, Batch Loss:     2.680154, Tokens per Sec:     2874, Lr: 0.000300\n",
      "2021-05-10 14:23:16,269 - INFO - joeynmt.training - Epoch   1, Step:     7800, Batch Loss:     2.449150, Tokens per Sec:     2816, Lr: 0.000300\n",
      "2021-05-10 14:24:46,025 - INFO - joeynmt.training - Epoch   1, Step:     7900, Batch Loss:     2.619170, Tokens per Sec:     2859, Lr: 0.000300\n",
      "2021-05-10 14:26:13,974 - INFO - joeynmt.training - Epoch   1, Step:     8000, Batch Loss:     2.975786, Tokens per Sec:     2801, Lr: 0.000300\n",
      "2021-05-10 14:28:39,473 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 14:28:39,473 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 14:28:39,474 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 14:28:39,745 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 14:28:39,745 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 14:28:39,960 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 14:28:39,960 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 14:28:39,960 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 14:28:39,960 - INFO - joeynmt.training - \tHypothesis: In 1997 the girls were found in Britain .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tHypothesis: It is not to know what he did not know the Bible to do the purpose of him and to support his heart .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tHypothesis: The main of the city in Sang , which is the Japanese African African , as the Europe of the Without of the human Hiberia , the process of the bank of their head .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - \tHypothesis: Explain and many deeds .\n",
      "2021-05-10 14:28:39,961 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     8000: bleu:  16.10, loss: 72340.8125, ppl:  12.4556, duration: 145.9870s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 14:30:08,639 - INFO - joeynmt.training - Epoch   1, Step:     8100, Batch Loss:     2.968779, Tokens per Sec:     2873, Lr: 0.000300\n",
      "2021-05-10 14:31:37,651 - INFO - joeynmt.training - Epoch   1, Step:     8200, Batch Loss:     2.754746, Tokens per Sec:     2891, Lr: 0.000300\n",
      "2021-05-10 14:33:06,065 - INFO - joeynmt.training - Epoch   1, Step:     8300, Batch Loss:     2.761967, Tokens per Sec:     2803, Lr: 0.000300\n",
      "2021-05-10 14:34:32,892 - INFO - joeynmt.training - Epoch   1, Step:     8400, Batch Loss:     2.588269, Tokens per Sec:     2832, Lr: 0.000300\n",
      "2021-05-10 14:35:59,909 - INFO - joeynmt.training - Epoch   1, Step:     8500, Batch Loss:     2.817042, Tokens per Sec:     2794, Lr: 0.000300\n",
      "2021-05-10 14:37:27,611 - INFO - joeynmt.training - Epoch   1, Step:     8600, Batch Loss:     2.819373, Tokens per Sec:     2837, Lr: 0.000300\n",
      "2021-05-10 14:38:55,587 - INFO - joeynmt.training - Epoch   1, Step:     8700, Batch Loss:     2.464084, Tokens per Sec:     2885, Lr: 0.000300\n",
      "2021-05-10 14:40:22,589 - INFO - joeynmt.training - Epoch   1, Step:     8800, Batch Loss:     2.556808, Tokens per Sec:     2890, Lr: 0.000300\n",
      "2021-05-10 14:41:51,488 - INFO - joeynmt.training - Epoch   1, Step:     8900, Batch Loss:     2.726137, Tokens per Sec:     2857, Lr: 0.000300\n",
      "2021-05-10 14:43:18,468 - INFO - joeynmt.training - Epoch   1, Step:     9000, Batch Loss:     2.456280, Tokens per Sec:     2796, Lr: 0.000300\n",
      "2021-05-10 14:46:08,948 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 14:46:08,948 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 14:46:08,948 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 14:46:09,219 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 14:46:09,220 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 14:46:09,434 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 14:46:09,434 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 14:46:09,434 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 14:46:09,434 - INFO - joeynmt.training - \tHypothesis: In 1997 a 90,000 girls found in England .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tHypothesis: It is not necessary to know what the Bible does not give him a purpose and support his own .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tHypothesis: The major of the city in Sweddg , which is the African African of the European African , the rejected the sleep to find their cope .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - \tHypothesis: Regarding the most sight and many stims .\n",
      "2021-05-10 14:46:09,435 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     9000: bleu:  17.35, loss: 69847.0703, ppl:  11.4184, duration: 170.9666s\n",
      "2021-05-10 14:47:37,866 - INFO - joeynmt.training - Epoch   1, Step:     9100, Batch Loss:     2.603883, Tokens per Sec:     2818, Lr: 0.000300\n",
      "2021-05-10 14:49:05,107 - INFO - joeynmt.training - Epoch   1, Step:     9200, Batch Loss:     2.455597, Tokens per Sec:     2793, Lr: 0.000300\n",
      "2021-05-10 14:50:31,955 - INFO - joeynmt.training - Epoch   1, Step:     9300, Batch Loss:     2.480252, Tokens per Sec:     2832, Lr: 0.000300\n",
      "2021-05-10 14:52:00,087 - INFO - joeynmt.training - Epoch   1, Step:     9400, Batch Loss:     3.095553, Tokens per Sec:     2828, Lr: 0.000300\n",
      "2021-05-10 14:53:27,841 - INFO - joeynmt.training - Epoch   1, Step:     9500, Batch Loss:     2.538290, Tokens per Sec:     2862, Lr: 0.000300\n",
      "2021-05-10 14:54:56,418 - INFO - joeynmt.training - Epoch   1, Step:     9600, Batch Loss:     2.212147, Tokens per Sec:     2854, Lr: 0.000300\n",
      "2021-05-10 14:56:24,308 - INFO - joeynmt.training - Epoch   1, Step:     9700, Batch Loss:     2.546114, Tokens per Sec:     2843, Lr: 0.000300\n",
      "2021-05-10 14:57:52,008 - INFO - joeynmt.training - Epoch   1, Step:     9800, Batch Loss:     2.591657, Tokens per Sec:     2845, Lr: 0.000300\n",
      "2021-05-10 14:57:58,465 - INFO - joeynmt.training - Epoch   1: total training loss 33437.65\n",
      "2021-05-10 14:57:58,465 - INFO - joeynmt.training - EPOCH 2\n",
      "2021-05-10 14:59:21,986 - INFO - joeynmt.training - Epoch   2, Step:     9900, Batch Loss:     2.833395, Tokens per Sec:     2753, Lr: 0.000300\n",
      "2021-05-10 15:00:49,888 - INFO - joeynmt.training - Epoch   2, Step:    10000, Batch Loss:     3.190526, Tokens per Sec:     2788, Lr: 0.000300\n",
      "2021-05-10 15:03:36,352 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 15:03:36,352 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 15:03:36,352 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 15:03:36,627 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 15:03:36,627 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 15:03:36,909 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tHypothesis: In 1997 the girls found in England .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tHypothesis: It is wise to tell him what the Bible does not give him to be a personal purpose and support to him .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tHypothesis: The chief in Strtrang , which is the headquarters of the European European African Empire , they returned to the philosophy to their copies .\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 15:03:36,910 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 15:03:36,911 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 15:03:36,911 - INFO - joeynmt.training - \tHypothesis: Remember , it is many susputes and many .\n",
      "2021-05-10 15:03:36,911 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    10000: bleu:  18.33, loss: 67433.7500, ppl:  10.4969, duration: 167.0217s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 15:05:06,882 - INFO - joeynmt.training - Epoch   2, Step:    10100, Batch Loss:     2.488953, Tokens per Sec:     2826, Lr: 0.000300\n",
      "2021-05-10 15:06:34,363 - INFO - joeynmt.training - Epoch   2, Step:    10200, Batch Loss:     2.318348, Tokens per Sec:     2865, Lr: 0.000300\n",
      "2021-05-10 15:08:02,181 - INFO - joeynmt.training - Epoch   2, Step:    10300, Batch Loss:     2.836575, Tokens per Sec:     2819, Lr: 0.000300\n",
      "2021-05-10 15:09:30,532 - INFO - joeynmt.training - Epoch   2, Step:    10400, Batch Loss:     2.896009, Tokens per Sec:     2831, Lr: 0.000300\n",
      "2021-05-10 15:10:59,365 - INFO - joeynmt.training - Epoch   2, Step:    10500, Batch Loss:     2.383335, Tokens per Sec:     2843, Lr: 0.000300\n",
      "2021-05-10 15:12:27,112 - INFO - joeynmt.training - Epoch   2, Step:    10600, Batch Loss:     2.260209, Tokens per Sec:     2839, Lr: 0.000300\n",
      "2021-05-10 15:13:55,346 - INFO - joeynmt.training - Epoch   2, Step:    10700, Batch Loss:     2.334218, Tokens per Sec:     2861, Lr: 0.000300\n",
      "2021-05-10 15:15:24,312 - INFO - joeynmt.training - Epoch   2, Step:    10800, Batch Loss:     2.322378, Tokens per Sec:     2834, Lr: 0.000300\n",
      "2021-05-10 15:16:52,043 - INFO - joeynmt.training - Epoch   2, Step:    10900, Batch Loss:     2.467571, Tokens per Sec:     2859, Lr: 0.000300\n",
      "2021-05-10 15:18:19,948 - INFO - joeynmt.training - Epoch   2, Step:    11000, Batch Loss:     2.245156, Tokens per Sec:     2850, Lr: 0.000300\n",
      "2021-05-10 15:20:43,292 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 15:20:43,292 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 15:20:43,293 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 15:20:43,562 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 15:20:43,562 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 15:20:43,775 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 15:20:43,775 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 15:20:43,775 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 15:20:43,775 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in Britain .\n",
      "2021-05-10 15:20:43,775 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tHypothesis: It is to know what the Bible does not be able to be satisfied by his suffering .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tHypothesis: The main statement in the city of Straskg , which is the headquarters of Europe as the Europe of Human Hiberia , the traveling the car to find their copy .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - \tHypothesis: Remember the most and many snakes .\n",
      "2021-05-10 15:20:43,776 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    11000: bleu:  20.09, loss: 65427.5938, ppl:   9.7878, duration: 143.8283s\n",
      "2021-05-10 15:22:12,435 - INFO - joeynmt.training - Epoch   2, Step:    11100, Batch Loss:     2.718374, Tokens per Sec:     2868, Lr: 0.000300\n",
      "2021-05-10 15:23:41,694 - INFO - joeynmt.training - Epoch   2, Step:    11200, Batch Loss:     2.804248, Tokens per Sec:     2851, Lr: 0.000300\n",
      "2021-05-10 15:25:10,117 - INFO - joeynmt.training - Epoch   2, Step:    11300, Batch Loss:     2.421386, Tokens per Sec:     2777, Lr: 0.000300\n",
      "2021-05-10 15:26:38,795 - INFO - joeynmt.training - Epoch   2, Step:    11400, Batch Loss:     2.607991, Tokens per Sec:     2847, Lr: 0.000300\n",
      "2021-05-10 15:28:07,104 - INFO - joeynmt.training - Epoch   2, Step:    11500, Batch Loss:     2.418192, Tokens per Sec:     2840, Lr: 0.000300\n",
      "2021-05-10 15:29:34,190 - INFO - joeynmt.training - Epoch   2, Step:    11600, Batch Loss:     2.103862, Tokens per Sec:     2794, Lr: 0.000300\n",
      "2021-05-10 15:31:04,670 - INFO - joeynmt.training - Epoch   2, Step:    11700, Batch Loss:     2.689956, Tokens per Sec:     2789, Lr: 0.000300\n",
      "2021-05-10 15:32:35,154 - INFO - joeynmt.training - Epoch   2, Step:    11800, Batch Loss:     2.534465, Tokens per Sec:     2737, Lr: 0.000300\n",
      "2021-05-10 15:34:04,194 - INFO - joeynmt.training - Epoch   2, Step:    11900, Batch Loss:     2.762211, Tokens per Sec:     2842, Lr: 0.000300\n",
      "2021-05-10 15:35:32,880 - INFO - joeynmt.training - Epoch   2, Step:    12000, Batch Loss:     2.664960, Tokens per Sec:     2834, Lr: 0.000300\n",
      "2021-05-10 15:38:04,341 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 15:38:04,341 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 15:38:04,341 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 15:38:04,611 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 15:38:04,611 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in England .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - \tHypothesis: It is not to tell him what the Bible does not understand what the purpose is to be satisfied by his suffering .\n",
      "2021-05-10 15:38:04,827 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tHypothesis: The main provision in the city of Straskourg , which is the home of the European European European Egyptian , traveling the branch office to find their copies .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - \tHypothesis: Remember and many global .\n",
      "2021-05-10 15:38:04,828 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    12000: bleu:  20.97, loss: 63807.8203, ppl:   9.2504, duration: 151.9473s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 15:39:33,410 - INFO - joeynmt.training - Epoch   2, Step:    12100, Batch Loss:     2.577723, Tokens per Sec:     2830, Lr: 0.000300\n",
      "2021-05-10 15:41:01,376 - INFO - joeynmt.training - Epoch   2, Step:    12200, Batch Loss:     2.754146, Tokens per Sec:     2821, Lr: 0.000300\n",
      "2021-05-10 15:42:30,303 - INFO - joeynmt.training - Epoch   2, Step:    12300, Batch Loss:     2.252576, Tokens per Sec:     2756, Lr: 0.000300\n",
      "2021-05-10 15:45:30,749 - INFO - joeynmt.training - Epoch   2, Step:    12500, Batch Loss:     2.649587, Tokens per Sec:     2761, Lr: 0.000300\n",
      "2021-05-10 15:46:58,825 - INFO - joeynmt.training - Epoch   2, Step:    12600, Batch Loss:     2.472306, Tokens per Sec:     2825, Lr: 0.000300\n",
      "2021-05-10 15:48:27,559 - INFO - joeynmt.training - Epoch   2, Step:    12700, Batch Loss:     2.329885, Tokens per Sec:     2807, Lr: 0.000300\n",
      "2021-05-10 15:49:54,806 - INFO - joeynmt.training - Epoch   2, Step:    12800, Batch Loss:     2.543615, Tokens per Sec:     2880, Lr: 0.000300\n",
      "2021-05-10 15:51:22,837 - INFO - joeynmt.training - Epoch   2, Step:    12900, Batch Loss:     2.080272, Tokens per Sec:     2884, Lr: 0.000300\n",
      "2021-05-10 15:52:50,526 - INFO - joeynmt.training - Epoch   2, Step:    13000, Batch Loss:     2.337162, Tokens per Sec:     2832, Lr: 0.000300\n",
      "2021-05-10 15:54:59,833 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 15:54:59,833 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 15:54:59,833 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 15:55:00,103 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 15:55:00,103 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 15:55:00,323 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in England .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tHypothesis: The wisdom must have to know what the Bible does not be able to be supported by his support .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - \tHypothesis: The main source in the city of Strasbourg , which is the home of the European Empire , traveling the room to find their copy .\n",
      "2021-05-10 15:55:00,324 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 15:55:00,325 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 15:55:00,325 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 15:55:00,325 - INFO - joeynmt.training - \tHypothesis: Remember the most and the global .\n",
      "2021-05-10 15:55:00,325 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    13000: bleu:  21.96, loss: 62265.4727, ppl:   8.7661, duration: 129.7984s\n",
      "2021-05-10 15:56:28,716 - INFO - joeynmt.training - Epoch   2, Step:    13100, Batch Loss:     2.274951, Tokens per Sec:     2856, Lr: 0.000300\n",
      "2021-05-10 15:57:55,581 - INFO - joeynmt.training - Epoch   2, Step:    13200, Batch Loss:     2.450541, Tokens per Sec:     2881, Lr: 0.000300\n",
      "2021-05-10 15:59:22,906 - INFO - joeynmt.training - Epoch   2, Step:    13300, Batch Loss:     2.340995, Tokens per Sec:     2854, Lr: 0.000300\n",
      "2021-05-10 16:00:49,584 - INFO - joeynmt.training - Epoch   2, Step:    13400, Batch Loss:     2.622322, Tokens per Sec:     2860, Lr: 0.000300\n",
      "2021-05-10 16:02:17,038 - INFO - joeynmt.training - Epoch   2, Step:    13500, Batch Loss:     2.419177, Tokens per Sec:     2798, Lr: 0.000300\n",
      "2021-05-10 16:03:44,857 - INFO - joeynmt.training - Epoch   2, Step:    13600, Batch Loss:     2.278650, Tokens per Sec:     2827, Lr: 0.000300\n",
      "2021-05-10 16:06:40,473 - INFO - joeynmt.training - Epoch   2, Step:    13800, Batch Loss:     2.539600, Tokens per Sec:     2853, Lr: 0.000300\n",
      "2021-05-10 16:08:07,836 - INFO - joeynmt.training - Epoch   2, Step:    13900, Batch Loss:     2.500989, Tokens per Sec:     2913, Lr: 0.000300\n",
      "2021-05-10 16:09:35,796 - INFO - joeynmt.training - Epoch   2, Step:    14000, Batch Loss:     2.492006, Tokens per Sec:     2889, Lr: 0.000300\n",
      "2021-05-10 16:11:46,710 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 16:11:46,711 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 16:11:46,711 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 16:11:46,978 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 16:11:46,978 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found the village in Britain .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tHypothesis: He must have to know what the Bible does not be able to be supported by his support .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 16:11:47,193 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - \tHypothesis: The main process in the city of Strasbourg , which is the North of Human Earth , traveling the cities to prevent their copy .\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - \tHypothesis: Remember the most and many disasters .\n",
      "2021-05-10 16:11:47,194 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    14000: bleu:  22.92, loss: 60852.6367, ppl:   8.3447, duration: 131.3972s\n",
      "2021-05-10 16:13:14,425 - INFO - joeynmt.training - Epoch   2, Step:    14100, Batch Loss:     2.560148, Tokens per Sec:     2873, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 16:14:42,256 - INFO - joeynmt.training - Epoch   2, Step:    14200, Batch Loss:     2.448101, Tokens per Sec:     2893, Lr: 0.000300\n",
      "2021-05-10 16:16:09,749 - INFO - joeynmt.training - Epoch   2, Step:    14300, Batch Loss:     2.667431, Tokens per Sec:     2778, Lr: 0.000300\n",
      "2021-05-10 16:17:37,716 - INFO - joeynmt.training - Epoch   2, Step:    14400, Batch Loss:     2.711628, Tokens per Sec:     2855, Lr: 0.000300\n",
      "2021-05-10 16:19:05,637 - INFO - joeynmt.training - Epoch   2, Step:    14500, Batch Loss:     2.636516, Tokens per Sec:     2828, Lr: 0.000300\n",
      "2021-05-10 16:20:32,133 - INFO - joeynmt.training - Epoch   2, Step:    14600, Batch Loss:     2.328991, Tokens per Sec:     2846, Lr: 0.000300\n",
      "2021-05-10 16:23:28,431 - INFO - joeynmt.training - Epoch   2, Step:    14800, Batch Loss:     2.399189, Tokens per Sec:     2859, Lr: 0.000300\n",
      "2021-05-10 16:24:57,288 - INFO - joeynmt.training - Epoch   2, Step:    14900, Batch Loss:     2.430764, Tokens per Sec:     2848, Lr: 0.000300\n",
      "2021-05-10 16:26:25,022 - INFO - joeynmt.training - Epoch   2, Step:    15000, Batch Loss:     2.095859, Tokens per Sec:     2831, Lr: 0.000300\n",
      "2021-05-10 16:28:39,402 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 16:28:39,403 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 16:28:39,403 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 16:28:39,671 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 16:28:39,672 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 16:28:39,887 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in England .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tHypothesis: He must have to know what the Bible does not be satisfied by his support .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tHypothesis: The chief in Strasbourg , which is the Attttalian Empire of Human Everyal European Empire , traveling to the cost of their copies .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - \tHypothesis: Remember the most and many diseases .\n",
      "2021-05-10 16:28:39,888 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    15000: bleu:  23.56, loss: 59718.6797, ppl:   8.0213, duration: 134.8659s\n",
      "2021-05-10 16:30:05,941 - INFO - joeynmt.training - Epoch   2, Step:    15100, Batch Loss:     2.266094, Tokens per Sec:     2854, Lr: 0.000300\n",
      "2021-05-10 16:31:34,194 - INFO - joeynmt.training - Epoch   2, Step:    15200, Batch Loss:     2.239600, Tokens per Sec:     2892, Lr: 0.000300\n",
      "2021-05-10 16:33:02,664 - INFO - joeynmt.training - Epoch   2, Step:    15300, Batch Loss:     2.434514, Tokens per Sec:     2833, Lr: 0.000300\n",
      "2021-05-10 16:34:30,637 - INFO - joeynmt.training - Epoch   2, Step:    15400, Batch Loss:     2.177247, Tokens per Sec:     2840, Lr: 0.000300\n",
      "2021-05-10 16:35:59,152 - INFO - joeynmt.training - Epoch   2, Step:    15500, Batch Loss:     2.103582, Tokens per Sec:     2836, Lr: 0.000300\n",
      "2021-05-10 16:37:29,214 - INFO - joeynmt.training - Epoch   2, Step:    15600, Batch Loss:     2.441991, Tokens per Sec:     2850, Lr: 0.000300\n",
      "2021-05-10 16:38:56,462 - INFO - joeynmt.training - Epoch   2, Step:    15700, Batch Loss:     2.284941, Tokens per Sec:     2816, Lr: 0.000300\n",
      "2021-05-10 16:40:22,745 - INFO - joeynmt.training - Epoch   2, Step:    15800, Batch Loss:     2.363835, Tokens per Sec:     2843, Lr: 0.000300\n",
      "2021-05-10 16:41:49,011 - INFO - joeynmt.training - Epoch   2, Step:    15900, Batch Loss:     2.183062, Tokens per Sec:     2849, Lr: 0.000300\n",
      "2021-05-10 16:45:28,244 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 16:45:28,244 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 16:45:28,244 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 16:45:28,518 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 16:45:28,518 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found a room in England .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible does not be removed by his support .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 16:45:28,733 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - \tHypothesis: The main stories in the Strasbourg , which is the headquarters of Human Empire , traveling the philosophy to find their copy .\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - \tHypothesis: Remember the many gardens and many garden .\n",
      "2021-05-10 16:45:28,734 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    16000: bleu:  23.75, loss: 58571.9453, ppl:   7.7069, duration: 131.5635s\n",
      "2021-05-10 16:46:56,038 - INFO - joeynmt.training - Epoch   2, Step:    16100, Batch Loss:     2.302176, Tokens per Sec:     2874, Lr: 0.000300\n",
      "2021-05-10 16:48:24,825 - INFO - joeynmt.training - Epoch   2, Step:    16200, Batch Loss:     2.276431, Tokens per Sec:     2817, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 16:49:51,930 - INFO - joeynmt.training - Epoch   2, Step:    16300, Batch Loss:     2.335480, Tokens per Sec:     2950, Lr: 0.000300\n",
      "2021-05-10 16:51:19,529 - INFO - joeynmt.training - Epoch   2, Step:    16400, Batch Loss:     2.371825, Tokens per Sec:     2872, Lr: 0.000300\n",
      "2021-05-10 16:52:47,261 - INFO - joeynmt.training - Epoch   2, Step:    16500, Batch Loss:     2.606483, Tokens per Sec:     2936, Lr: 0.000300\n",
      "2021-05-10 16:54:14,152 - INFO - joeynmt.training - Epoch   2, Step:    16600, Batch Loss:     2.476956, Tokens per Sec:     2838, Lr: 0.000300\n",
      "2021-05-10 16:55:42,195 - INFO - joeynmt.training - Epoch   2, Step:    16700, Batch Loss:     2.273109, Tokens per Sec:     2872, Lr: 0.000300\n",
      "2021-05-10 16:57:09,085 - INFO - joeynmt.training - Epoch   2, Step:    16800, Batch Loss:     2.470606, Tokens per Sec:     2869, Lr: 0.000300\n",
      "2021-05-10 16:58:35,336 - INFO - joeynmt.training - Epoch   2, Step:    16900, Batch Loss:     2.239506, Tokens per Sec:     2818, Lr: 0.000300\n",
      "2021-05-10 17:00:03,333 - INFO - joeynmt.training - Epoch   2, Step:    17000, Batch Loss:     2.247704, Tokens per Sec:     2860, Lr: 0.000300\n",
      "2021-05-10 17:02:11,842 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 17:02:11,842 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 17:02:11,842 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 17:02:12,110 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 17:02:12,110 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 17:02:12,325 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found a room in Britain .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tHypothesis: He must have to know what the Bible says is not to be intensed by his support .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - \tHypothesis: The main statement in the Strasbourg , which is the Strasbourg of the European Empire , traveling the philosophy to prevent their copy .\n",
      "2021-05-10 17:02:12,326 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 17:02:12,327 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 17:02:12,327 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 17:02:12,327 - INFO - joeynmt.training - \tHypothesis: Remember the most and the garden .\n",
      "2021-05-10 17:02:12,327 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    17000: bleu:  24.84, loss: 57695.2266, ppl:   7.4749, duration: 128.9931s\n",
      "2021-05-10 17:03:39,036 - INFO - joeynmt.training - Epoch   2, Step:    17100, Batch Loss:     1.949176, Tokens per Sec:     2899, Lr: 0.000300\n",
      "2021-05-10 17:05:05,646 - INFO - joeynmt.training - Epoch   2, Step:    17200, Batch Loss:     2.030777, Tokens per Sec:     2879, Lr: 0.000300\n",
      "2021-05-10 17:06:32,176 - INFO - joeynmt.training - Epoch   2, Step:    17300, Batch Loss:     2.125240, Tokens per Sec:     2926, Lr: 0.000300\n",
      "2021-05-10 17:07:58,502 - INFO - joeynmt.training - Epoch   2, Step:    17400, Batch Loss:     2.137732, Tokens per Sec:     2883, Lr: 0.000300\n",
      "2021-05-10 17:09:26,339 - INFO - joeynmt.training - Epoch   2, Step:    17500, Batch Loss:     2.221572, Tokens per Sec:     2884, Lr: 0.000300\n",
      "2021-05-10 17:10:53,583 - INFO - joeynmt.training - Epoch   2, Step:    17600, Batch Loss:     2.523746, Tokens per Sec:     2848, Lr: 0.000300\n",
      "2021-05-10 17:15:17,676 - INFO - joeynmt.training - Epoch   2, Step:    17900, Batch Loss:     2.180850, Tokens per Sec:     2890, Lr: 0.000300\n",
      "2021-05-10 17:16:44,157 - INFO - joeynmt.training - Epoch   2, Step:    18000, Batch Loss:     2.076793, Tokens per Sec:     2882, Lr: 0.000300\n",
      "2021-05-10 17:18:57,613 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 17:18:57,613 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 17:18:57,614 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 17:18:57,880 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 17:18:57,881 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found room in England .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible does not be removed by his support .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 17:18:58,094 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - \tHypothesis: The main center in the city of Strasbourg , which is the Atttecular European Earth , traveling the traveling of the catch to prevent their copies .\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - \tHypothesis: Remember the most and discreet .\n",
      "2021-05-10 17:18:58,095 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    18000: bleu:  25.14, loss: 56873.7656, ppl:   7.2638, duration: 133.9372s\n",
      "2021-05-10 17:20:24,511 - INFO - joeynmt.training - Epoch   2, Step:    18100, Batch Loss:     2.152098, Tokens per Sec:     2841, Lr: 0.000300\n",
      "2021-05-10 17:21:50,812 - INFO - joeynmt.training - Epoch   2, Step:    18200, Batch Loss:     2.139853, Tokens per Sec:     2891, Lr: 0.000300\n",
      "2021-05-10 17:23:19,087 - INFO - joeynmt.training - Epoch   2, Step:    18300, Batch Loss:     2.258672, Tokens per Sec:     2930, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 17:24:47,373 - INFO - joeynmt.training - Epoch   2, Step:    18400, Batch Loss:     2.140292, Tokens per Sec:     2894, Lr: 0.000300\n",
      "2021-05-10 17:26:15,494 - INFO - joeynmt.training - Epoch   2, Step:    18500, Batch Loss:     2.206281, Tokens per Sec:     2896, Lr: 0.000300\n",
      "2021-05-10 17:27:40,942 - INFO - joeynmt.training - Epoch   2, Step:    18600, Batch Loss:     2.277930, Tokens per Sec:     2827, Lr: 0.000300\n",
      "2021-05-10 17:29:06,849 - INFO - joeynmt.training - Epoch   2, Step:    18700, Batch Loss:     2.116841, Tokens per Sec:     2892, Lr: 0.000300\n",
      "2021-05-10 17:30:35,663 - INFO - joeynmt.training - Epoch   2, Step:    18800, Batch Loss:     2.076362, Tokens per Sec:     2846, Lr: 0.000300\n",
      "2021-05-10 17:32:03,173 - INFO - joeynmt.training - Epoch   2, Step:    18900, Batch Loss:     2.053634, Tokens per Sec:     2866, Lr: 0.000300\n",
      "2021-05-10 17:33:30,952 - INFO - joeynmt.training - Epoch   2, Step:    19000, Batch Loss:     2.275691, Tokens per Sec:     2897, Lr: 0.000300\n",
      "2021-05-10 17:35:38,865 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 17:35:38,866 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 17:35:38,866 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 17:35:39,133 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 17:35:39,134 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 17:35:39,348 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found a room in England .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible says to the purpose of his support .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the home of the European Earth , the travelers arranged to prevent their copies .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 17:35:39,349 - INFO - joeynmt.training - \tHypothesis: Remember the most and disaster .\n",
      "2021-05-10 17:35:39,350 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step    19000: bleu:  25.92, loss: 55897.6758, ppl:   7.0208, duration: 128.3968s\n",
      "2021-05-10 17:37:06,138 - INFO - joeynmt.training - Epoch   2, Step:    19100, Batch Loss:     2.810566, Tokens per Sec:     2927, Lr: 0.000300\n",
      "2021-05-10 17:38:33,979 - INFO - joeynmt.training - Epoch   2, Step:    19200, Batch Loss:     2.189923, Tokens per Sec:     2885, Lr: 0.000300\n",
      "2021-05-10 17:40:00,161 - INFO - joeynmt.training - Epoch   2, Step:    19300, Batch Loss:     2.141145, Tokens per Sec:     2877, Lr: 0.000300\n",
      "2021-05-10 17:41:27,555 - INFO - joeynmt.training - Epoch   2, Step:    19400, Batch Loss:     2.023566, Tokens per Sec:     2906, Lr: 0.000300\n",
      "2021-05-10 17:42:54,143 - INFO - joeynmt.training - Epoch   2, Step:    19500, Batch Loss:     2.116561, Tokens per Sec:     2875, Lr: 0.000300\n",
      "2021-05-10 17:44:20,606 - INFO - joeynmt.training - Epoch   2, Step:    19600, Batch Loss:     2.195735, Tokens per Sec:     2904, Lr: 0.000300\n",
      "2021-05-10 17:44:29,698 - INFO - joeynmt.training - Epoch   2: total training loss 23096.25\n",
      "2021-05-10 17:44:29,698 - INFO - joeynmt.training - EPOCH 3\n",
      "2021-05-10 17:45:49,613 - INFO - joeynmt.training - Epoch   3, Step:    19700, Batch Loss:     2.022207, Tokens per Sec:     2840, Lr: 0.000300\n",
      "2021-05-10 17:47:19,247 - INFO - joeynmt.training - Epoch   3, Step:    19800, Batch Loss:     2.395367, Tokens per Sec:     2829, Lr: 0.000300\n",
      "2021-05-10 17:48:48,192 - INFO - joeynmt.training - Epoch   3, Step:    19900, Batch Loss:     2.234133, Tokens per Sec:     2889, Lr: 0.000300\n",
      "2021-05-10 17:50:14,263 - INFO - joeynmt.training - Epoch   3, Step:    20000, Batch Loss:     2.041868, Tokens per Sec:     2825, Lr: 0.000300\n",
      "2021-05-10 17:52:25,177 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 17:52:25,178 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 17:52:25,178 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 17:52:25,443 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 17:52:25,443 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 17:52:25,656 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found a room in England .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible says to be convinced and support to him .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the Art of Human Earth , the travelers arranged to prevent their copy .\n",
      "2021-05-10 17:52:25,657 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 17:52:25,658 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 17:52:25,658 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 17:52:25,658 - INFO - joeynmt.training - \tHypothesis: Remember the most and discreet .\n",
      "2021-05-10 17:52:25,658 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    20000: bleu:  26.10, loss: 55378.2656, ppl:   6.8948, duration: 131.3945s\n",
      "2021-05-10 17:53:52,081 - INFO - joeynmt.training - Epoch   3, Step:    20100, Batch Loss:     2.382305, Tokens per Sec:     2905, Lr: 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 17:55:19,466 - INFO - joeynmt.training - Epoch   3, Step:    20200, Batch Loss:     2.271461, Tokens per Sec:     2915, Lr: 0.000300\n",
      "2021-05-10 17:56:47,625 - INFO - joeynmt.training - Epoch   3, Step:    20300, Batch Loss:     2.046909, Tokens per Sec:     2876, Lr: 0.000300\n",
      "2021-05-10 17:58:14,182 - INFO - joeynmt.training - Epoch   3, Step:    20400, Batch Loss:     2.206532, Tokens per Sec:     2864, Lr: 0.000300\n",
      "2021-05-10 17:59:41,190 - INFO - joeynmt.training - Epoch   3, Step:    20500, Batch Loss:     2.080265, Tokens per Sec:     2907, Lr: 0.000300\n",
      "2021-05-10 18:01:09,133 - INFO - joeynmt.training - Epoch   3, Step:    20600, Batch Loss:     2.179455, Tokens per Sec:     2891, Lr: 0.000300\n",
      "2021-05-10 18:02:36,323 - INFO - joeynmt.training - Epoch   3, Step:    20700, Batch Loss:     2.198697, Tokens per Sec:     2881, Lr: 0.000300\n",
      "2021-05-10 18:04:04,127 - INFO - joeynmt.training - Epoch   3, Step:    20800, Batch Loss:     2.115926, Tokens per Sec:     2858, Lr: 0.000300\n",
      "2021-05-10 18:05:30,260 - INFO - joeynmt.training - Epoch   3, Step:    20900, Batch Loss:     2.086459, Tokens per Sec:     2882, Lr: 0.000300\n",
      "2021-05-10 18:07:00,744 - INFO - joeynmt.training - Epoch   3, Step:    21000, Batch Loss:     2.107676, Tokens per Sec:     2659, Lr: 0.000300\n",
      "2021-05-10 18:09:41,202 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 18:09:41,203 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 18:09:41,203 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 18:09:41,470 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 18:09:41,471 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 18:09:41,682 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found the room in England .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible says to be removed by his support .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the European Court of Human Rights , the travelers arranged to produce their copy .\n",
      "2021-05-10 18:09:41,683 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 18:09:41,684 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 18:09:41,684 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 18:09:41,684 - INFO - joeynmt.training - \tHypothesis: Remember the most and discreet .\n",
      "2021-05-10 18:09:41,684 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    21000: bleu:  26.24, loss: 55252.9492, ppl:   6.8647, duration: 160.9396s\n",
      "2021-05-10 18:11:09,026 - INFO - joeynmt.training - Epoch   3, Step:    21100, Batch Loss:     1.908183, Tokens per Sec:     2904, Lr: 0.000300\n",
      "2021-05-10 18:12:36,192 - INFO - joeynmt.training - Epoch   3, Step:    21200, Batch Loss:     2.253291, Tokens per Sec:     2901, Lr: 0.000300\n",
      "2021-05-10 18:14:03,734 - INFO - joeynmt.training - Epoch   3, Step:    21300, Batch Loss:     2.119918, Tokens per Sec:     2896, Lr: 0.000300\n",
      "2021-05-10 18:15:30,836 - INFO - joeynmt.training - Epoch   3, Step:    21400, Batch Loss:     1.739385, Tokens per Sec:     2875, Lr: 0.000300\n",
      "2021-05-10 18:16:59,084 - INFO - joeynmt.training - Epoch   3, Step:    21500, Batch Loss:     2.232154, Tokens per Sec:     2892, Lr: 0.000300\n",
      "2021-05-10 18:18:25,922 - INFO - joeynmt.training - Epoch   3, Step:    21600, Batch Loss:     2.261061, Tokens per Sec:     2901, Lr: 0.000300\n",
      "2021-05-10 18:19:53,332 - INFO - joeynmt.training - Epoch   3, Step:    21700, Batch Loss:     2.158993, Tokens per Sec:     2851, Lr: 0.000300\n",
      "2021-05-10 18:21:24,054 - INFO - joeynmt.training - Epoch   3, Step:    21800, Batch Loss:     2.268159, Tokens per Sec:     2785, Lr: 0.000300\n",
      "2021-05-10 18:22:51,171 - INFO - joeynmt.training - Epoch   3, Step:    21900, Batch Loss:     2.183103, Tokens per Sec:     2904, Lr: 0.000300\n",
      "2021-05-10 18:24:16,550 - INFO - joeynmt.training - Epoch   3, Step:    22000, Batch Loss:     2.112784, Tokens per Sec:     2837, Lr: 0.000300\n",
      "2021-05-10 18:26:15,849 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 18:26:15,849 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 18:26:15,849 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 18:26:16,113 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 18:26:16,113 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in England .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible does not be convinced and support of him .\n",
      "2021-05-10 18:26:16,328 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the European homes of Human High , travelers arranged to predict their copy .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - \tHypothesis: Remember the most and discreet .\n",
      "2021-05-10 18:26:16,329 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    22000: bleu:  26.95, loss: 54241.4805, ppl:   6.6269, duration: 119.7793s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 18:27:42,816 - INFO - joeynmt.training - Epoch   3, Step:    22100, Batch Loss:     2.179298, Tokens per Sec:     2884, Lr: 0.000300\n",
      "2021-05-10 18:29:09,642 - INFO - joeynmt.training - Epoch   3, Step:    22200, Batch Loss:     1.911644, Tokens per Sec:     2919, Lr: 0.000300\n",
      "2021-05-10 18:30:36,544 - INFO - joeynmt.training - Epoch   3, Step:    22300, Batch Loss:     2.098760, Tokens per Sec:     2874, Lr: 0.000300\n",
      "2021-05-10 18:32:02,872 - INFO - joeynmt.training - Epoch   3, Step:    22400, Batch Loss:     2.001364, Tokens per Sec:     2844, Lr: 0.000300\n",
      "2021-05-10 18:33:29,896 - INFO - joeynmt.training - Epoch   3, Step:    22500, Batch Loss:     1.917742, Tokens per Sec:     2880, Lr: 0.000300\n",
      "2021-05-10 18:34:56,473 - INFO - joeynmt.training - Epoch   3, Step:    22600, Batch Loss:     1.971986, Tokens per Sec:     2844, Lr: 0.000300\n",
      "2021-05-10 18:36:24,135 - INFO - joeynmt.training - Epoch   3, Step:    22700, Batch Loss:     2.091261, Tokens per Sec:     2924, Lr: 0.000300\n",
      "2021-05-10 18:37:51,747 - INFO - joeynmt.training - Epoch   3, Step:    22800, Batch Loss:     2.165265, Tokens per Sec:     2887, Lr: 0.000300\n",
      "2021-05-10 18:39:17,249 - INFO - joeynmt.training - Epoch   3, Step:    22900, Batch Loss:     1.974013, Tokens per Sec:     2896, Lr: 0.000300\n",
      "2021-05-10 18:40:44,745 - INFO - joeynmt.training - Epoch   3, Step:    23000, Batch Loss:     2.112074, Tokens per Sec:     2881, Lr: 0.000300\n",
      "2021-05-10 18:42:45,172 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 18:42:45,172 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 18:42:45,172 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 18:42:45,436 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 18:42:45,436 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found a room in Britain .\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 18:42:45,648 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible says to be convinced and support to him .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tHypothesis: At the main station in the city of Strasbourg , which is the Earth of Human Earth , the travelers arranged to prevent their copy .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - \tHypothesis: Remember the most and discreet .\n",
      "2021-05-10 18:42:45,649 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    23000: bleu:  27.01, loss: 53518.8750, ppl:   6.4620, duration: 120.9040s\n",
      "2021-05-10 18:44:12,188 - INFO - joeynmt.training - Epoch   3, Step:    23100, Batch Loss:     1.892363, Tokens per Sec:     2925, Lr: 0.000300\n",
      "2021-05-10 18:45:39,005 - INFO - joeynmt.training - Epoch   3, Step:    23200, Batch Loss:     2.022973, Tokens per Sec:     2962, Lr: 0.000300\n",
      "2021-05-10 18:47:05,902 - INFO - joeynmt.training - Epoch   3, Step:    23300, Batch Loss:     1.946127, Tokens per Sec:     2913, Lr: 0.000300\n",
      "2021-05-10 18:48:31,741 - INFO - joeynmt.training - Epoch   3, Step:    23400, Batch Loss:     2.462944, Tokens per Sec:     2797, Lr: 0.000300\n",
      "2021-05-10 18:49:58,563 - INFO - joeynmt.training - Epoch   3, Step:    23500, Batch Loss:     2.276049, Tokens per Sec:     2930, Lr: 0.000300\n",
      "2021-05-10 18:51:25,520 - INFO - joeynmt.training - Epoch   3, Step:    23600, Batch Loss:     1.861068, Tokens per Sec:     2884, Lr: 0.000300\n",
      "2021-05-10 18:52:52,909 - INFO - joeynmt.training - Epoch   3, Step:    23700, Batch Loss:     1.905351, Tokens per Sec:     2894, Lr: 0.000300\n",
      "2021-05-10 18:54:20,114 - INFO - joeynmt.training - Epoch   3, Step:    23800, Batch Loss:     2.171257, Tokens per Sec:     2803, Lr: 0.000300\n",
      "2021-05-10 18:55:47,172 - INFO - joeynmt.training - Epoch   3, Step:    23900, Batch Loss:     1.932601, Tokens per Sec:     2931, Lr: 0.000300\n",
      "2021-05-10 18:57:13,394 - INFO - joeynmt.training - Epoch   3, Step:    24000, Batch Loss:     2.071873, Tokens per Sec:     2837, Lr: 0.000300\n",
      "2021-05-10 18:59:14,344 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 18:59:14,344 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 18:59:14,344 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 18:59:14,606 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 18:59:14,607 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 18:59:14,822 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 18:59:14,822 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 18:59:14,822 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 18:59:14,822 - INFO - joeynmt.training - \tHypothesis: In 1997 90,000 girls found room in England .\n",
      "2021-05-10 18:59:14,822 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible does not be convinced and support to him .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the Pharisees of Human Empire , travelers arranged to predict their copy .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - \tHypothesis: Remember the abundance and discreet .\n",
      "2021-05-10 18:59:14,823 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    24000: bleu:  27.07, loss: 53311.3945, ppl:   6.4154, duration: 121.4290s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-10 19:00:40,537 - INFO - joeynmt.training - Epoch   3, Step:    24100, Batch Loss:     1.761699, Tokens per Sec:     2854, Lr: 0.000300\n",
      "2021-05-10 19:02:06,905 - INFO - joeynmt.training - Epoch   3, Step:    24200, Batch Loss:     1.903977, Tokens per Sec:     2923, Lr: 0.000300\n",
      "2021-05-10 19:03:32,696 - INFO - joeynmt.training - Epoch   3, Step:    24300, Batch Loss:     1.905390, Tokens per Sec:     2903, Lr: 0.000300\n",
      "2021-05-10 19:04:59,193 - INFO - joeynmt.training - Epoch   3, Step:    24400, Batch Loss:     2.093480, Tokens per Sec:     2926, Lr: 0.000300\n",
      "2021-05-10 19:06:25,690 - INFO - joeynmt.training - Epoch   3, Step:    24500, Batch Loss:     2.135311, Tokens per Sec:     2954, Lr: 0.000300\n",
      "2021-05-10 19:07:51,921 - INFO - joeynmt.training - Epoch   3, Step:    24600, Batch Loss:     2.122672, Tokens per Sec:     2913, Lr: 0.000300\n",
      "2021-05-10 19:09:18,775 - INFO - joeynmt.training - Epoch   3, Step:    24700, Batch Loss:     2.238747, Tokens per Sec:     2899, Lr: 0.000300\n",
      "2021-05-10 19:10:45,515 - INFO - joeynmt.training - Epoch   3, Step:    24800, Batch Loss:     1.964722, Tokens per Sec:     2877, Lr: 0.000300\n",
      "2021-05-10 19:12:11,358 - INFO - joeynmt.training - Epoch   3, Step:    24900, Batch Loss:     1.914498, Tokens per Sec:     2828, Lr: 0.000300\n",
      "2021-05-10 19:13:36,237 - INFO - joeynmt.training - Epoch   3, Step:    25000, Batch Loss:     1.887765, Tokens per Sec:     2859, Lr: 0.000300\n",
      "2021-05-10 19:15:43,509 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 19:15:43,509 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 19:15:43,509 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 19:15:43,771 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2021-05-10 19:15:43,771 - INFO - joeynmt.training - Saving new checkpoint.\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in England .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tHypothesis: You must have learned what the Bible says to be convinced and reliable to him .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - \tHypothesis: At the capital of Strasbourg , which is a Roman European Rights of Human Rights , travelers arranged to predict their copy .\n",
      "2021-05-10 19:15:43,985 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 19:15:43,986 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 19:15:43,986 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 19:15:43,986 - INFO - joeynmt.training - \tHypothesis: Remember many and discreet .\n",
      "2021-05-10 19:15:43,986 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    25000: bleu:  27.79, loss: 52673.2461, ppl:   6.2743, duration: 127.7482s\n",
      "2021-05-10 19:17:12,858 - INFO - joeynmt.training - Epoch   3, Step:    25100, Batch Loss:     1.952466, Tokens per Sec:     2839, Lr: 0.000300\n",
      "2021-05-10 19:18:39,812 - INFO - joeynmt.training - Epoch   3, Step:    25200, Batch Loss:     2.080122, Tokens per Sec:     2931, Lr: 0.000300\n",
      "2021-05-10 19:20:05,524 - INFO - joeynmt.training - Epoch   3, Step:    25300, Batch Loss:     2.109455, Tokens per Sec:     2961, Lr: 0.000300\n",
      "2021-05-10 19:21:31,503 - INFO - joeynmt.training - Epoch   3, Step:    25400, Batch Loss:     1.890165, Tokens per Sec:     2905, Lr: 0.000300\n",
      "2021-05-10 19:22:58,327 - INFO - joeynmt.training - Epoch   3, Step:    25500, Batch Loss:     1.891301, Tokens per Sec:     2929, Lr: 0.000300\n",
      "2021-05-10 19:24:23,619 - INFO - joeynmt.training - Epoch   3, Step:    25600, Batch Loss:     2.036573, Tokens per Sec:     2899, Lr: 0.000300\n",
      "2021-05-10 19:25:49,673 - INFO - joeynmt.training - Epoch   3, Step:    25700, Batch Loss:     1.813404, Tokens per Sec:     2892, Lr: 0.000300\n",
      "2021-05-10 19:27:15,983 - INFO - joeynmt.training - Epoch   3, Step:    25800, Batch Loss:     1.910212, Tokens per Sec:     2904, Lr: 0.000300\n",
      "2021-05-10 19:28:41,847 - INFO - joeynmt.training - Epoch   3, Step:    25900, Batch Loss:     2.352136, Tokens per Sec:     2895, Lr: 0.000300\n",
      "2021-05-10 19:30:07,490 - INFO - joeynmt.training - Epoch   3, Step:    26000, Batch Loss:     1.867678, Tokens per Sec:     2912, Lr: 0.000300\n",
      "2021-05-10 19:33:38,167 - INFO - joeynmt.training - Epoch   3, Step:    26100, Batch Loss:     2.038053, Tokens per Sec:     2964, Lr: 0.000300\n",
      "2021-05-10 19:35:03,751 - INFO - joeynmt.training - Epoch   3, Step:    26200, Batch Loss:     1.955604, Tokens per Sec:     2889, Lr: 0.000300\n",
      "2021-05-10 19:36:31,590 - INFO - joeynmt.training - Epoch   3, Step:    26300, Batch Loss:     1.804468, Tokens per Sec:     2940, Lr: 0.000300\n",
      "2021-05-10 19:37:56,751 - INFO - joeynmt.training - Epoch   3, Step:    26400, Batch Loss:     1.958242, Tokens per Sec:     2898, Lr: 0.000300\n",
      "2021-05-10 19:39:22,943 - INFO - joeynmt.training - Epoch   3, Step:    26500, Batch Loss:     2.022110, Tokens per Sec:     2905, Lr: 0.000300\n",
      "2021-05-10 19:40:49,341 - INFO - joeynmt.training - Epoch   3, Step:    26600, Batch Loss:     2.556990, Tokens per Sec:     2925, Lr: 0.000300\n",
      "2021-05-10 19:42:14,828 - INFO - joeynmt.training - Epoch   3, Step:    26700, Batch Loss:     2.087514, Tokens per Sec:     2881, Lr: 0.000300\n",
      "2021-05-10 19:43:41,227 - INFO - joeynmt.training - Epoch   3, Step:    26800, Batch Loss:     1.980137, Tokens per Sec:     2917, Lr: 0.000300\n",
      "2021-05-10 19:45:05,907 - INFO - joeynmt.training - Epoch   3, Step:    26900, Batch Loss:     1.983151, Tokens per Sec:     2885, Lr: 0.000300\n",
      "2021-05-10 19:46:31,410 - INFO - joeynmt.training - Epoch   3, Step:    27000, Batch Loss:     1.855923, Tokens per Sec:     2928, Lr: 0.000300\n",
      "2021-05-10 19:48:29,500 - WARNING - sacrebleu - That's 100 lines that end in a tokenized period ('.')\n",
      "2021-05-10 19:48:29,501 - WARNING - sacrebleu - It looks like you forgot to detokenize your test data, which may hurt your score.\n",
      "2021-05-10 19:48:29,501 - WARNING - sacrebleu - If you insist your data is detokenized, or don't care, you can suppress this message with '--force'.\n",
      "2021-05-10 19:48:29,960 - INFO - joeynmt.training - Example #0\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tHypothesis: In 1997 the 90,000 girls found in Britain .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - Example #1\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tHypothesis: You must have to know what the Bible says not to be convinced and reliable .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - Example #2\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tHypothesis: At the main center in the city of Strasbourg , which is the European Rights , travelers sought to find their copy .\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - Example #3\n",
      "2021-05-10 19:48:29,961 - INFO - joeynmt.training - \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2021-05-10 19:48:29,962 - INFO - joeynmt.training - \tReference:  Lots of hugs and kisses .\n",
      "2021-05-10 19:48:29,962 - INFO - joeynmt.training - \tHypothesis: Remember , and many discreet .\n",
      "2021-05-10 19:48:29,962 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    27000: bleu:  26.92, loss: 53394.1797, ppl:   6.4340, duration: 118.5513s\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_reverse_$tgt$src.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${tgt}${src}_reverse_transformer/* \"$gdrive_path/models/${src}${tgt}_reverse_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n94wlrCjVc17"
   },
   "outputs": [],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"$gdrive_path/models/${tgt}${src}_reverse_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "66WhRE9lIhoD"
   },
   "outputs": [],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${tgt}${src}_reverse_transformer/config.yaml\""
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "starter_notebook_reverse_training.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}