![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "accqQwSNN7-F"
+ },
+ "source": [
+ "# Fine-tune Spanish RoBERTa (BSC) for QA on SQAC / by [mrm8488](https://twitter.com/mrm8488)\n",
+ "\n",
+ "> Based on Sylvain Gugger [Colab](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ivCfi4TeCSas"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "VT8tUzVZB7Cs",
+ "outputId": "d19edc33-2974-40c5-9ce6-315b9f3e1a06"
+ },
+ "source": [
+ "!nvidia-smi"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Tue Oct 5 10:42:28 2021 \n",
+ "+-----------------------------------------------------------------------------+\n",
+ "| NVIDIA-SMI 470.74 Driver Version: 460.32.03 CUDA Version: 11.2 |\n",
+ "|-------------------------------+----------------------+----------------------+\n",
+ "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
+ "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
+ "| | | MIG M. |\n",
+ "|===============================+======================+======================|\n",
+ "| 0 A100-SXM4-40GB Off | 00000000:00:04.0 Off | 0 |\n",
+ "| N/A 32C P0 41W / 400W | 0MiB / 40536MiB | 0% Default |\n",
+ "| | | Disabled |\n",
+ "+-------------------------------+----------------------+----------------------+\n",
+ " \n",
+ "+-----------------------------------------------------------------------------+\n",
+ "| Processes: |\n",
+ "| GPU GI CI PID Type Process name GPU Memory |\n",
+ "| ID ID Usage |\n",
+ "|=============================================================================|\n",
+ "| No running processes found |\n",
+ "+-----------------------------------------------------------------------------+\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "H603Kk-CCDts",
+ "outputId": "89251286-4144-48cd-9248-b2865f7e96b4"
+ },
+ "source": [
+ "# ! pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
+ "Requirement already satisfied: torch==1.9.0+cu111 in /usr/local/lib/python3.7/dist-packages (1.9.0+cu111)\n",
+ "Requirement already satisfied: torchvision==0.10.0+cu111 in /usr/local/lib/python3.7/dist-packages (0.10.0+cu111)\n",
+ "Requirement already satisfied: torchaudio==0.9.0 in /usr/local/lib/python3.7/dist-packages (0.9.0)\n",
+ "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.9.0+cu111) (3.7.4.3)\n",
+ "Requirement already satisfied: pillow>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision==0.10.0+cu111) (7.1.2)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchvision==0.10.0+cu111) (1.19.5)\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "MOsHUjgdIrIW"
+ },
+ "source": [
+ "%%capture\n",
+ "! pip install datasets transformers"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4-1djuIFzH-U"
+ },
+ "source": [
+ "Si está abriendo este cuaderno localmente, asegúrese de que su entorno tenga una instalación de la última versión de esas bibliotecas.\n",
+ "\n",
+ "Para poder compartir su modelo con la comunidad y generar resultados como el que se muestra en la imagen a continuación a través de la API de inferencia, hay algunos pasos más a seguir.\n",
+ "\n",
+ "Primero debe almacenar su token de autenticación del sitio web Hugging Face (regístrese [aquí](https://huggingface.co/join) si aún no lo ha hecho). Luego ejecute la siguiente celda e ingrese su nombre de usuario y contraseña:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "vXGwn15pzH-U"
+ },
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "\n",
+ "notebook_login()"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PyerhFJHzH-V"
+ },
+ "source": [
+ "Necesitamos instalar Git-LFS. Descomenta y ejecuta la siguiente celda"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "M8Qzgd71zH-V"
+ },
+ "source": [
+ "! apt install git-lfs"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PMhSzj9kzH-W"
+ },
+ "source": [
+ "Asegúrese de que su versión de Transformers sea al menos 4.11.0 ya que la funcionalidad se introdujo en esa versión:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "m1dvKnz6zH-W",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "f16b7094-7f8d-4404-dc4b-06a66d10b495"
+ },
+ "source": [
+ "import transformers\n",
+ "\n",
+ "print(transformers.__version__)"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "4.11.2\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HFASsisvIrIb"
+ },
+ "source": [
+ "You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs [here](https://github.com/huggingface/transformers/tree/master/examples/question-answering)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rEJBSTyZIrIb"
+ },
+ "source": [
+ "# Fine-tuning de un modelo para la tarea de QA"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rFOqCFAwzH-X"
+ },
+ "source": [
+ "En este cuaderno, veremos cómo hacer \"fine-tuning\" a uno de los modelos de [🤗 Transformers](https://github.com/huggingface/transformers) para la tarea de respuesta a una pregunta (QA), que es la tarea de extraer la respuesta a una pregunta de un contexto dado. Veremos cómo cargar fácilmente un conjunto de datos para este tipo de tareas y usar la API `Trainer` para ajustar un modelo en él.\n",
+ "\n",
+ "\n",
+ "\n",
+ "** Nota: ** Este cuaderno afina los modelos que responden preguntas tomando una subcadena de un contexto, no generando texto nuevo."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "zVvslsfMIrIh"
+ },
+ "source": [
+ "squad_v2 = False\n",
+ "model_checkpoint = \"BSC-TeMU/roberta-base-bne\"\n",
+ "batch_size = 16"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "whPRbBNbIrIl"
+ },
+ "source": [
+ "## Cargando el dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "W7QYTpxXIrIl"
+ },
+ "source": [
+ "Usaremos la biblioteca [🤗 Datasets](https://github.com/huggingface/datasets) para descargar los datos y obtener la métrica que necesitamos usar para la evaluación (para comparar nuestro modelo con el benchmark). Esto se puede hacer fácilmente con las funciones `load_dataset` y` load_metric`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "IreSlFmlIrIm"
+ },
+ "source": [
+ "from datasets import load_dataset, load_metric"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "CKx2zKs5IrIq"
+ },
+ "source": [
+ "Para nuestro ejemplo usaremos el [Dataset SQAC](https://huggingface.co/datasets/BSC-TeMU/SQAC). El cuaderno debe funcionar con cualquier conjunto de datos de respuesta a preguntas proporcionado por la biblioteca 🤗 Datasets. Si está utilizando su propio conjunto de datos definido a partir de un archivo JSON o csv (consulte la [documentación de Datasets[texto del enlace](https://)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) sobre cómo cargarlos ), es posible que necesite algunos ajustes en los nombres de las columnas utilizadas."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 296,
+ "referenced_widgets": [
+ "8da92867c5b44e80ab7a2b4731ede590",
+ "20a81ea50f5f493182dafc57b0be2c4c",
+ "6e1e06162c7449b7bcc6a489d665aaaa",
+ "32a63b55ed1f4cc48d5c4209132929fa",
+ "317db374e85c49aeb67e7eee7ca9eb9e",
+ "1031b20b8d2748d386f40f71b091808a",
+ "a8b23c0c30a54acf99d205c40959dacb",
+ "fef9bc9e5826482897a0f0a164e4168a",
+ "43937bb9d5d944c3b030ff863d6aef80",
+ "f33bb66719f94d479e59a8b8a0ee7106",
+ "755e9028a4954dfa9a4775336be62040",
+ "29bfa26690dd43c5918180775b48937e",
+ "76d523e3477443d794d1c5692ccd9d38",
+ "c8f9450d7f6e4d90a11e8bcb088c2225",
+ "200d5ca959054446bc50e2cb4006dbc4",
+ "720aa191a19044efbddaf1039c45e8a9",
+ "54b93c05e0d342498020ad8dd0bfc351",
+ "8877c9db24364603803a7eca08236f05",
+ "51a17d0911064d54a3a67a492a88ec0e",
+ "f9b452e4da324032bde0144ae5f0022e",
+ "fbf1a1f5742d4bb9a5c93fbc64cbc464",
+ "7ce07165e8294016a7de941738ae8389",
+ "f1009e1822c14c59ab70039006d0f38e",
+ "affbb1f7d25d49029b13ae02c73668be",
+ "8aee3fa61b36413b8d80478c4be09243",
+ "eb7fe655be5d4aefb2a0c4c26a48796b",
+ "979ebd3e0db14655b8effb3fbcb336a1",
+ "a345f336c3dd458a88036aaa9fbc0d1f",
+ "6082998f17ce4f39b2df2ac3867e500e",
+ "f6865238c00041aba1b371b0a2c946d5",
+ "21ccf35dcd024568b8daa0c2a1e2b479",
+ "cf599c3d13104f44812d8a8b2f1784ce",
+ "4841318237ce49eab3d28843c335f4ed",
+ "bf3ef2d0c83f4242982163830f56b0bc",
+ "e2f7d510f6dc48cba86492836d55a935",
+ "d5619c4cf98742da8f9938d4a6c0b6d4",
+ "781fba40d21440458d6bcb3d3072484a",
+ "188c44a48a16429bb72e816ae164c9b3",
+ "9503d51d35234c86861d704fb66feb91",
+ "27076bf4de4045d9b4058921b52f91e0",
+ "a57abe4411d248269d190f5af32880ca",
+ "1d2e0967076a42e1a8c8bde745cc35d3",
+ "16d7bfb41a8341bcbf9cf359fc97a345",
+ "ae7786adb5e14e8592b1309483280e72",
+ "0cd9c9f81dde43a7b0011b4609b01a5c",
+ "416dc21ddac34d5abee1d293ce4f5931",
+ "a34d5b521df5429bb78ef1cd85e39362",
+ "c2f718a827674a43a765f772fc67b3d1",
+ "62240abb9caf423a8acbf0785b656fe9",
+ "1c263f80958d4bb2aa8141aad2cbe0d5",
+ "64641b95c0384bf5bf1f8d2a94e2686f",
+ "dd1e24749ec64c3184698acb25700e98",
+ "cb2617281cc94fcd98b2288941a64a99",
+ "62bd5b1222904823a5dd3ba419d1387e",
+ "66d2d3c701a049e38caf3cbcd19092ba",
+ "0241437e79d04f81be6968166795d323",
+ "ece8003b7d7a4741ad111c525ac8b41a",
+ "9c9052fc37f24b3ca554c8e9e8741e2c",
+ "30c5046e64e2415f8f12a6b40399fe2c",
+ "2dfd9d7856a24ea3a0d62839a38c98d8",
+ "26d9b443f5b94b71b2b111dc3748efdb",
+ "2d049d96dd5547f0a5885e058fdad375",
+ "25a6c1634c554967a617e5646ecd81c6",
+ "46385133fe9f4a1b99ec077db3aec2b6",
+ "ddf7a78c31ea4faea0b1c7a3e38594b6",
+ "19a3dac1145c417b8aa0444bd046196c",
+ "f170c233f06e48239c1b180956ab097e",
+ "3585870ff79e40faa96a25361c84529b",
+ "75ec9e40729549c192d4f231e457ed85",
+ "5fb18fc7ee214a85a2058170100302af",
+ "71d06d73e5ec4fb29b26c5cdf3060b44",
+ "0659f1faa05440a8b27be7c07e967ce9",
+ "2c7deb2f80af42578c4c95f8415ee543",
+ "a5376d15a8ed41c3aaf60dcae932f147",
+ "3b76a94d44054e1383e92efd1fe6a0fa",
+ "7e1dd612a8284dbf8a54e23341971edd",
+ "52117fc88c7e4972b1051cb7ffcf32e3",
+ "c80dd3c6d4894a50835263372c65b5bf",
+ "360178e85d424251823add25e6971275",
+ "4e58a6f8517e45d1821adcf5ce7ff416",
+ "0a01025b58a843339777741a0f055088",
+ "f396653f40394a0aa242e54e88ca47fc",
+ "ab838d93f88c43c2b92a9f7f2761c67d",
+ "f2d56bed0410420ca29d707ee5442c8a",
+ "3d8b544c4d7f4cf2b26ac80e09b9b997",
+ "03cee110bcfe440bb3f0660ac935f080",
+ "1a55d593e7a84b6aad88915d24db93d0",
+ "a031a8a789784f8ca7e7b708fb0e9a7b",
+ "20774b700c9d4669b4cf7af87b22ce29",
+ "b97012a8fad34ffc9eae05adb260c018",
+ "5b35e716d5fa49e0a3f17ceacafa9ef2",
+ "578f4c280ae945dcb266551cf72df3f4",
+ "cc978b75fc0d48fbbae205c39afebc65",
+ "439edf682f8848a5b444900e4d51cb94",
+ "df8b76c9657b46e794ce224563d81344",
+ "817687c1baf9464288516376b8ad2e0a",
+ "f308364939a84955ba7a8ca81c6808d1",
+ "4e06bfcb009e456b800ca6fa3e0d93e4",
+ "996f8177c10941c3aee55b57615a8329",
+ "2f9017a010d54278abf177d2d554e8fa",
+ "135b344aa24f416099dbadfa6ad9070f",
+ "3499dd4eb4824588a1cb7a9bca74064f",
+ "dbac1955c39143bbaf0da37cd2d12384",
+ "a554026728854d928ff7ebe0a59fce77",
+ "2eaa764656a44eca9d4a8b533dedc696",
+ "44acb3acb7444de1a24c20e3702c3c70",
+ "a03da758ba57490e8ef75067a0c1c11b",
+ "31f45890df624d58b58f6f5a53504767",
+ "4eacee5cc37349928d89c36972f3330e",
+ "c47c066ad5c14218ac55acd190e0a16b"
+ ]
+ },
+ "id": "s_AY1ATSIrIq",
+ "outputId": "1b4d496b-b573-4c61-a2c7-a0c4fdc758d4"
+ },
+ "source": [
+ "datasets = load_dataset(\"BSC-TeMU/SQAC\")"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "8da92867c5b44e80ab7a2b4731ede590",
+ "version_minor": 0,
+ "version_major": 2
+ },
+ "text/plain": [
+ "Downloading: 0%| | 0.00/5.56k [00:00\n",
+ " \n",
+ " | Epoch | \n", + "Training Loss | \n", + "Validation Loss | \n", + "
|---|---|---|
| 1 | \n", + "0.997100 | \n", + "0.864572 | \n", + "
| 2 | \n", + "0.482000 | \n", + "0.933443 | \n", + "
| 3 | \n", + "0.165200 | \n", + "1.211125 | \n", + "
"
+ ],
+ "text/plain": [
+ "