en-af/main/elan_en_af_masakhane_jw300.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "elan_en_af_masakhane.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to [email protected]\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "e4cf5e6a-6683-4011-c4bf-9db4355b1774",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# ! echo \"4/sAFT6Lt4k6urQdvj0H6vghJt7kl6X9dCG0gd-XY_1mMDstx7QiAJ0qM\" | python -c \"from google.colab import drive; drive.mount('/content/drive')\""
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"af\"\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "gdrive_path = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)\n",
        "os.environ[\"gdrive_path\"] = gdrive_path\n",
        "!mkdir -p \"$gdrive_path\"\n",
        "# !mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "ab1167ec-3654-4667-af5a-57f4ed16aca5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path\n",
        "# !ls $gdrive_path"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-af-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "ab59d363-609c-4a36-b3aa-e9e0f5de80e4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/e7/005433050cf3d76bca0fbbc59631ce493fdabb029bbd795a08f26003a6bd/opustools_pkg-0.0.50-py3-none-any.whl (49kB)\n",
            "\r\u001b[K     |██████▋                         | 10kB 17.4MB/s eta 0:00:01\r\u001b[K     |█████████████▏                  | 20kB 1.8MB/s eta 0:00:01\r\u001b[K     |███████████████████▉            | 30kB 2.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▍     | 40kB 1.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 51kB 2.0MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.50\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "outputId": "1b6f3ba5-a887-4753-d0f7-630a5cd75b20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# change working directory\n",
        "os.chdir(gdrive_path)\n",
        "\n",
        "# Download our corpus\n",
        "! if ! (( test -f jw300.$src ) && ( test -f jw300.$tgt )); then opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q; else echo \"Opus files are already present, skipping download.\"; fi\n",
        "\n",
        "# extract the corpus file\n",
        "! if test -f JW300_latest_xml_$src-$tgt.xml.gz; then gunzip JW300_latest_xml_$src-$tgt.xml.gz; elif test -f JW300_latest_xml_$tgt-$src.xml.gz; then gunzip JW300_latest_xml_$tgt-$src.xml.gz; else echo \"ERROR: missing corpus file!\"; fi"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Opus files are already present, skipping download.\n",
            "ERROR: missing corpus file!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "a442231a-8f44-4d68-98bb-f7fb9cd98c5b"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "with open(source_file) as f:\n",
        "  for _, line in enumerate(f):\n",
        "    source.append(line)\n",
        "with open(target_file) as f:\n",
        "  for _, line in enumerate(f):\n",
        "    target.append(line)\n",
        "\n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>‘ They Lifted Me Out of Deep Depression ’\\n</td>\n",
              "      <td>‘ Hulle het my uit diep depressie gehelp ’\\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A woman from England wrote :\\n</td>\n",
              "      <td>’ n Vrou van Engeland het geskryf :\\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>“ Dear Sir :\\n</td>\n",
              "      <td>“ Geagte Meneer :\\n</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                               source_sentence                               target_sentence\n",
              "0  ‘ They Lifted Me Out of Deep Depression ’\\n  ‘ Hulle het my uit diep depressie gehelp ’\\n\n",
              "1               A woman from England wrote :\\n         ’ n Vrou van Engeland het geskryf :\\n\n",
              "2                               “ Dear Sir :\\n                           “ Geagte Meneer :\\n"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "9606b898-616a-4c0b-eeb3-658828253a73"
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
            "  after removing the cwd from sys.path.\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
            "  \"\"\"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "7b4a4141-be14-4c89-9e81-627b6d164717"
      },
      "source": [
        "# This section does the split between train/test/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and 1000 test set. In practice, it's useful to use an external test set\n",
        "\n",
        "# Do the split between dev/test/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "num_test_patterns = 1000\n",
        "\n",
        "# Lower case the corpora\n",
        "df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "devtest = df_pp.tail(num_dev_patterns + num_test_patterns)\n",
        "test = devtest.tail(num_test_patterns) # Herman\n",
        "dev = devtest.head(num_dev_patterns)  # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns + num_test_patterns).index)\n",
        "\n",
        "stripped[[\"source_sentence\"]].to_csv(\"train.en\", index=False)\n",
        "stripped[[\"target_sentence\"]].to_csv(\"train.af\", index=False)\n",
        "\n",
        "dev[[\"source_sentence\"]].to_csv(\"dev.en\", index=False)\n",
        "dev[[\"target_sentence\"]].to_csv(\"dev.af\", index=False)\n",
        "\n",
        "test[[\"source_sentence\"]].to_csv(\"test.en\", index=False)\n",
        "test[[\"target_sentence\"]].to_csv(\"test.af\", index=False)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
            "  \"\"\"\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
            "  \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6fc6dbc6-99a3-4ac2-f3b4-eb4e543954c0"
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "\n",
        "# swap to branch that has the \"ElanScheduler\"! and install that :D\n",
        "# ! cd joeynmt; git checkout scheduler; git pull --all; pip3 install . # pip install --upgrade --force-reinstall --no-deps <package>\n",
        "! cd joeynmt; git checkout symlink; git pull --all; pip3 install . # pip install --upgrade --force-reinstall --no-deps <package>\n",
        "\n",
        "# perform default installation\n",
        "# ! cd joeynmt; git checkout master; git pull --all; pip3 install ."
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'joeynmt' already exists and is not an empty directory.\n",
            "M\tscripts/generate_copy_task.py\n",
            "M\tscripts/generate_reverse_task.py\n",
            "M\tscripts/get_iwslt14_bpe.sh\n",
            "M\tscripts/get_iwslt15_envi.sh\n",
            "M\tscripts/plot_validations.py\n",
            "Already on 'symlink'\n",
            "Your branch is up to date with 'origin/symlink'.\n",
            "Fetching origin\n",
            "Already up to date.\n",
            "Processing /content/drive/My Drive/masakhane/en-af-baseline/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (4.3.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.16.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (41.2.0)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.2.0)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0rc3)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6 (from joeynmt==0.0.1)\n",
            "  Downloading https://files.pythonhosted.org/packages/0e/e5/93d252182f7cbd4b59bb3ec5797e2ce33cfd6f5aadaf327db170cf4b7887/sacrebleu-1.4.2-py3-none-any.whl\n",
            "Collecting subword-nmt (from joeynmt==0.0.1)\n",
            "  Downloading https://files.pythonhosted.org/packages/26/08/58267cb3ac00f5f895457777ed9e0d106dbb5e6388fa7923d8663b04b849/subword_nmt-0.3.6-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.0.3)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
            "Collecting pyyaml>=5.1 (from joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)\n",
            "\u001b[K     |████████████████████████████████| 266kB 7.5MB/s \n",
            "\u001b[?25hCollecting pylint (from joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ef/ed/1cb8e7b85a31807aa0bff8b3e60935370bed7e141df8b530aac6352bddff/pylint-2.4.2-py3-none-any.whl (302kB)\n",
            "\u001b[K     |████████████████████████████████| 307kB 39.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: six>=1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->joeynmt==0.0.1) (0.46)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.7.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.7)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
            "Collecting portalocker (from sacrebleu>=1.3.6->joeynmt==0.0.1)\n",
            "  Downloading https://files.pythonhosted.org/packages/60/ec/836a494dbaa72541f691ec4e66f29fdc8db9bcc7f49e1c2d457ba13ced42/portalocker-1.5.1-py2.py3-none-any.whl\n",
            "Collecting typing (from sacrebleu>=1.3.6->joeynmt==0.0.1)\n",
            "  Downloading https://files.pythonhosted.org/packages/fe/2e/b480ee1b75e6d17d2993738670e75c1feeb9ff7f64452153cf018051cc92/typing-3.7.4.1-py3-none-any.whl\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.5.3)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.2)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.3.1)\n",
            "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.24.2)\n",
            "Collecting isort<5,>=4.2.5 (from pylint->joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 21.8MB/s \n",
            "\u001b[?25hCollecting mccabe<0.7,>=0.6 (from pylint->joeynmt==0.0.1)\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Collecting astroid<2.4,>=2.3.0 (from pylint->joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/13/e1/74a63c85c501c29c52da5be604c025e368f4dd77daf1fa13c878a33e5a36/astroid-2.3.1-py3-none-any.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 41.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.9.11)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\" (from astroid<2.4,>=2.3.0->pylint->joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/31/d3/9d1802c161626d0278bafb1ffb32f76b9d01e123881bbf9d91e8ccf28e18/typed_ast-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (736kB)\n",
            "\u001b[K     |████████████████████████████████| 737kB 41.6MB/s \n",
            "\u001b[?25hCollecting lazy-object-proxy==1.4.* (from astroid<2.4,>=2.3.0->pylint->joeynmt==0.0.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0e/26/534a6d32572a9dbca11619321535c0a7ab34688545d9d67c2c204b9e3a3d/lazy_object_proxy-1.4.2-cp36-cp36m-manylinux1_x86_64.whl (49kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 13.9MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=69462 sha256=2eb100ac13e868b7e68943631f0ba842d5d081a3f1dee88932c5fd0189fd4217\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-8heq9clf/wheels/4c/ba/65/529ea2efaa773ebbedd9df75a20586f3366c4efa375e38c09c\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=e97131de73fdca9b5d0c399df8ab27c8ccfaababae68ad7562d7cdb481e12cce\n",
            "  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030\n",
            "Successfully built joeynmt pyyaml\n",
            "Installing collected packages: portalocker, typing, sacrebleu, subword-nmt, pyyaml, isort, mccabe, typed-ast, lazy-object-proxy, astroid, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed astroid-2.3.1 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.2 mccabe-0.6.1 portalocker-1.5.1 pylint-2.4.2 pyyaml-5.1.2 sacrebleu-1.4.2 subword-nmt-0.3.6 typed-ast-1.4.0 typing-3.7.4.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        },
        "outputId": "fe8692cb-96d2-4a5f-dc36-48d2607bbbf7"
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "# from os import path\n",
        "\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman!\n",
        "\n",
        "! if ! (( test -f vocab.$src ) && ( test -f vocab.$tgt )); then subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt; else echo \"BPE vocab files already present...\"; fi\n",
        "\n",
        "! if ! test -f train.bpe.$src; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src; else echo \"train.bpe.$src already present...\"; fi\n",
        "! if ! test -f train.bpe.$tgt; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt; else echo \"train.bpe.$tgt already present...\"; fi\n",
        "\n",
        "! if ! test -f dev.bpe.$src; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src; else echo \"dev.bpe.$src already present...\"; fi\n",
        "! if ! test -f dev.bpe.$tgt; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt; else echo \"dev.bpe.$tgt already present...\"; fi\n",
        "\n",
        "! if ! test -f test.bpe.$src; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src; else echo \"test.bpe.$src already present...\"; fi\n",
        "! if ! test -f test.bpe.$tgt; then subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt; else echo \"test.bpe.$tgt already present...\"; fi\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! if ! test $(ls -l $data_path | grep train | wc -l) -gt 0; then cp train.* $data_path; else echo \"train files already in data directory...\"; fi\n",
        "! if ! test $(ls -l $data_path | grep test | wc -l) -gt 0; then cp test.* $data_path; else echo \"test files already in data directory...\"; fi\n",
        "! if ! test $(ls -l $data_path | grep dev | wc -l) -gt 0; then cp dev.* $data_path; else echo \"dev files already in data directory...\"; fi\n",
        "! if ! test -f $data_path/bpe.codes.4000; then cp bpe.codes.4000 $data_path ; else echo \"bpe.codes.4000 already in data directory...\"; fi\n",
        "! ls $data_path\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! if ! test -f joeynmt/data/$src$tgt/vocab.txt; then joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt ; else echo \"vocab.txt already in data directory...\"; fi\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Afrikaans Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "BPE vocab files already present...\n",
            "train.bpe.en already present...\n",
            "train.bpe.af already present...\n",
            "dev.bpe.en already present...\n",
            "dev.bpe.af already present...\n",
            "test.bpe.en already present...\n",
            "test.bpe.af already present...\n",
            "train files already in data directory...\n",
            "test files already in data directory...\n",
            "dev files already in data directory...\n",
            "bpe.codes.4000 already in data directory...\n",
            "bpe.codes.4000\tdev.bpe.en  test.bpe.af  train.af      train.en\n",
            "dev.af\t\tdev.en\t    test.bpe.en  train.bpe.af  vocab.txt\n",
            "dev.bpe.af\ttest.af     test.en\t train.bpe.en\n",
            "vocab.txt already in data directory...\n",
            "BPE Afrikaans Sentences\n",
            "\"\n",
            "\"maar ons kan daarvan seker wees dat ons eer@@ likheid en ander goeie eienskappe vir ons hemelse vader baie ko@@ sb@@ aar@@ der is as enige ed@@ el@@ ste@@ en !\n",
            "\"\n",
            "\"as ons eer@@ lik is , het ons ’ n sk@@ oon gewe@@ te en vryheid van spraak in die bediening\n",
            "\"\n",
            "Combined BPE Vocab\n",
            "·@@\n",
            "̣@@\n",
            "̈@@\n",
            "§\n",
            "ז@@\n",
            "inst\n",
            "˜\n",
            ";@@\n",
            "ḥ\n",
            "saja\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# # This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# # (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "# name = '%s%s' % (source_language, target_language)\n",
        "# gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# # Create the config\n",
        "# config = \"\"\"\n",
        "# name: \"{name}_transformer\"\n",
        "\n",
        "# data:\n",
        "#     src: \"{source_language}\"\n",
        "#     trg: \"{target_language}\"\n",
        "#     train: \"data/{name}/train.bpe\"\n",
        "#     dev:   \"data/{name}/dev.bpe\"\n",
        "#     test:  \"data/{name}/test.bpe\"\n",
        "#     level: \"bpe\"\n",
        "#     lowercase: False\n",
        "#     max_sent_length: 100\n",
        "#     src_vocab: \"data/{name}/vocab.txt\"\n",
        "#     trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "# testing:\n",
        "#     beam_size: 5\n",
        "#     alpha: 1.0\n",
        "\n",
        "# training:\n",
        "#     #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "#     random_seed: 42\n",
        "#     optimizer: \"adam\"\n",
        "#     normalization: \"tokens\"\n",
        "#     adam_betas: [0.9, 0.999] \n",
        "#     scheduling: \"elan\"            # Try switching to Elan scheduling\n",
        "#     learning_rate_decay_length: 5000 # number of steps to reduce by the decay factor for Elan method\n",
        "#     learning_rate_peak: 0.002  # peak for Elan scheduler (default: 1)\n",
        "#     learning_rate_warmup: 2000  # warmup steps for Elan scheduler\n",
        "#     learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "#     learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "#     patience: 8\n",
        "#     decrease_factor: 0.7\n",
        "#     loss: \"crossentropy\"\n",
        "#     learning_rate: 0.0002\n",
        "#     learning_rate_min: 0.00000001\n",
        "#     weight_decay: 0.0\n",
        "#     label_smoothing: 0.1\n",
        "#     batch_size: 4096\n",
        "#     batch_type: \"token\"\n",
        "#     eval_batch_size: 3600\n",
        "#     eval_batch_type: \"token\"\n",
        "#     batch_multiplier: 1\n",
        "#     early_stopping_metric: \"ppl\"\n",
        "#     epochs: 30 # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "#     validation_freq: 500 # 4000 # Decrease this for testing\n",
        "#     logging_freq: 100\n",
        "#     eval_metric: \"bleu\"\n",
        "#     model_dir: \"models/{name}_transformer\"\n",
        "#     overwrite: True\n",
        "#     shuffle: True\n",
        "#     use_cuda: True\n",
        "#     max_output_length: 100\n",
        "#     print_valid_sents: [0, 1, 2, 3]\n",
        "#     keep_last_ckpts: 3\n",
        "\n",
        "# model:\n",
        "#     initializer: \"xavier\"\n",
        "#     bias_initializer: \"zeros\"\n",
        "#     init_gain: 1.0\n",
        "#     embed_initializer: \"xavier\"\n",
        "#     embed_init_gain: 1.0\n",
        "#     tied_embeddings: True\n",
        "#     tied_softmax: True\n",
        "#     encoder:\n",
        "#         type: \"transformer\"\n",
        "#         num_layers: 3\n",
        "#         num_heads: 8\n",
        "#         embeddings:\n",
        "#             embedding_dim: 512\n",
        "#             scale: True\n",
        "#             dropout: 0.\n",
        "#         # typically ff_size = 4 x hidden_size\n",
        "#         hidden_size: 512\n",
        "#         ff_size: 2048\n",
        "#         dropout: 0.3\n",
        "#     decoder:\n",
        "#         type: \"transformer\"\n",
        "#         num_layers: 3\n",
        "#         num_heads: 8\n",
        "#         embeddings:\n",
        "#             embedding_dim: 512\n",
        "#             scale: True\n",
        "#             dropout: 0.\n",
        "#         # typically ff_size = 4 x hidden_size\n",
        "#         hidden_size: 512\n",
        "#         ff_size: 2048\n",
        "#         dropout: 0.25\n",
        "# \"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "# with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "#     f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "luNmAL42JDd7",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"data/{name}/train.bpe\"\n",
        "    dev:   \"data/{name}/dev.bpe\"\n",
        "    test:  \"data/{name}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"data/{name}/vocab.txt\"\n",
        "    trg_vocab: \"data/{name}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    #load_model: \"{gdrive_path}/joeynmt/models/{name}_transformer/3500.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    load_model: \"{gdrive_path}/Backup_models/Copy of best.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"noam\"            # Try switching to Elan scheduling\n",
        "    learning_rate_decay_length: 5000 # number of steps to reduce by the decay factor for Elan method\n",
        "    learning_rate_peak: 0.002  # peak for Elan scheduler (default: 1)\n",
        "    learning_rate_warmup: 2000  # warmup steps for Elan scheduler\n",
        "    learning_rate_factor: 1       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    patience: 8\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0002\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 30 # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 500 # 4000 # Decrease this for testing\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: True\n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 3\n",
        "        num_heads: 8\n",
        "        embeddings:\n",
        "            embedding_dim: 512\n",
        "            scale: True\n",
        "            dropout: 0.\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 512\n",
        "        ff_size: 2048\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 3\n",
        "        num_heads: 8\n",
        "        embeddings:\n",
        "            embedding_dim: 512\n",
        "            scale: True\n",
        "            dropout: 0.\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 512\n",
        "        ff_size: 2048\n",
        "        dropout: 0.25\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(os.path.join(gdrive_path, f\"joeynmt/configs/transformer_{name}.yaml\"),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "colab": {}
      },
      "source": [
        "# # Train the model\n",
        "# # You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "\n",
        "# os.chdir(os.path.join(gdrive_path, \"joeynmt\"))\n",
        "\n",
        "# ! python3 joeynmt/__main__.py train configs/transformer_$src$tgt.yaml\n",
        "# # !cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "colab": {}
      },
      "source": [
        "# Copy the created models from the notebook storage to google drive for persistant storage \n",
        "# !cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\"\n",
        "! mv \"$gdrive_path/../../masakhane_models_backup/models\" \"$gdrive_path/joeynmt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "outputId": "72875a99-fbd7-4518-9d7d-a7deda40f3fc"
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 500\tLoss: 90178.07812\tPPL: 37.62176\tbleu: 0.90432\tLR: 0.00069877\t*\n",
            "Steps: 1000\tLoss: 73574.21094\tPPL: 19.29143\tbleu: 2.83208\tLR: 0.00139754\t*\n",
            "Steps: 1500\tLoss: 67591.24219\tPPL: 15.16492\tbleu: 4.42870\tLR: 0.00114109\t*\n",
            "Steps: 2000\tLoss: 63030.15234\tPPL: 12.62282\tbleu: 5.49023\tLR: 0.00098821\t*\n",
            "Steps: 2500\tLoss: 58729.75781\tPPL: 10.61762\tbleu: 7.85880\tLR: 0.00088388\t*\n",
            "Steps: 3000\tLoss: 54868.16016\tPPL: 9.09000\tbleu: 10.19263\tLR: 0.00080687\t*\n",
            "Steps: 3500\tLoss: 51913.49219\tPPL: 8.07133\tbleu: 11.93531\tLR: 0.00074702\t*\n",
            "Steps: 4000\tLoss: 48699.18750\tPPL: 7.09235\tbleu: 15.10315\tLR: 0.00069877\t*\n",
            "Steps: 4500\tLoss: 46169.38281\tPPL: 6.40610\tbleu: 17.91368\tLR: 0.00065881\t*\n",
            "Steps: 5000\tLoss: 43576.91016\tPPL: 5.77168\tbleu: 21.04279\tLR: 0.00062500\t*\n",
            "Steps: 5500\tLoss: 40994.23438\tPPL: 5.20214\tbleu: 23.94864\tLR: 0.00059591\t*\n",
            "Steps: 6000\tLoss: 38960.73438\tPPL: 4.79354\tbleu: 26.45850\tLR: 0.00057054\t*\n",
            "Steps: 6500\tLoss: 36407.32422\tPPL: 4.32561\tbleu: 30.21616\tLR: 0.00054816\t*\n",
            "Steps: 7000\tLoss: 34931.22266\tPPL: 4.07624\tbleu: 31.84504\tLR: 0.00052822\t*\n",
            "Steps: 7500\tLoss: 33079.28125\tPPL: 3.78360\tbleu: 34.36116\tLR: 0.00051031\t*\n",
            "Steps: 8000\tLoss: 31651.22070\tPPL: 3.57237\tbleu: 35.70833\tLR: 0.00049411\t*\n",
            "Steps: 8500\tLoss: 30685.96289\tPPL: 3.43632\tbleu: 37.68450\tLR: 0.00047935\t*\n",
            "Steps: 9000\tLoss: 29720.62891\tPPL: 3.30544\tbleu: 38.99970\tLR: 0.00046585\t*\n",
            "Steps: 9500\tLoss: 28773.71875\tPPL: 3.18189\tbleu: 41.25283\tLR: 0.00045342\t*\n",
            "Steps: 10000\tLoss: 28502.89258\tPPL: 3.14742\tbleu: 40.88431\tLR: 0.00044194\t*\n",
            "Steps: 10500\tLoss: 28264.90039\tPPL: 3.11743\tbleu: 41.76735\tLR: 0.00043129\t*\n",
            "Steps: 11000\tLoss: 27241.58398\tPPL: 2.99171\tbleu: 43.46731\tLR: 0.00042137\t*\n",
            "Steps: 11500\tLoss: 26806.92188\tPPL: 2.93985\tbleu: 43.37800\tLR: 0.00041211\t*\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "c139f283-db05-404d-c239-87d68bb46595"
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-10-16 13:50:44,139 -  dev bleu:  45.10 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2019-10-16 13:51:41,319 - test bleu:  45.48 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}