![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "accqQwSNN7-F"
},
"source": [
"# Fine-tune Spanish RoBERTa (BSC) for QA on SQAC / by [mrm8488](https://twitter.com/mrm8488)\n",
"\n",
"> Based on Sylvain Gugger [Colab](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ivCfi4TeCSas"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VT8tUzVZB7Cs",
"outputId": "d19edc33-2974-40c5-9ce6-315b9f3e1a06"
},
"source": [
"!nvidia-smi"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Tue Oct 5 10:42:28 2021 \n",
"+-----------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 470.74 Driver Version: 460.32.03 CUDA Version: 11.2 |\n",
"|-------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|===============================+======================+======================|\n",
"| 0 A100-SXM4-40GB Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 32C P0 41W / 400W | 0MiB / 40536MiB | 0% Default |\n",
"| | | Disabled |\n",
"+-------------------------------+----------------------+----------------------+\n",
" \n",
"+-----------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=============================================================================|\n",
"| No running processes found |\n",
"+-----------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "H603Kk-CCDts",
"outputId": "89251286-4144-48cd-9248-b2865f7e96b4"
},
"source": [
"# ! pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
"Requirement already satisfied: torch==1.9.0+cu111 in /usr/local/lib/python3.7/dist-packages (1.9.0+cu111)\n",
"Requirement already satisfied: torchvision==0.10.0+cu111 in /usr/local/lib/python3.7/dist-packages (0.10.0+cu111)\n",
"Requirement already satisfied: torchaudio==0.9.0 in /usr/local/lib/python3.7/dist-packages (0.9.0)\n",
"Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.9.0+cu111) (3.7.4.3)\n",
"Requirement already satisfied: pillow>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision==0.10.0+cu111) (7.1.2)\n",
"Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchvision==0.10.0+cu111) (1.19.5)\n"
]
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "MOsHUjgdIrIW"
},
"source": [
"%%capture\n",
"! pip install datasets transformers"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "4-1djuIFzH-U"
},
"source": [
"Si está abriendo este cuaderno localmente, asegúrese de que su entorno tenga una instalación de la última versión de esas bibliotecas.\n",
"\n",
"Para poder compartir su modelo con la comunidad y generar resultados como el que se muestra en la imagen a continuación a través de la API de inferencia, hay algunos pasos más a seguir.\n",
"\n",
"Primero debe almacenar su token de autenticación del sitio web Hugging Face (regístrese [aquí](https://huggingface.co/join) si aún no lo ha hecho). Luego ejecute la siguiente celda e ingrese su nombre de usuario y contraseña:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "vXGwn15pzH-U"
},
"source": [
"from huggingface_hub import notebook_login\n",
"\n",
"notebook_login()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "PyerhFJHzH-V"
},
"source": [
"Necesitamos instalar Git-LFS. Descomenta y ejecuta la siguiente celda"
]
},
{
"cell_type": "code",
"metadata": {
"id": "M8Qzgd71zH-V"
},
"source": [
"! apt install git-lfs"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "PMhSzj9kzH-W"
},
"source": [
"Asegúrese de que su versión de Transformers sea al menos 4.11.0 ya que la funcionalidad se introdujo en esa versión:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "m1dvKnz6zH-W",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "f16b7094-7f8d-4404-dc4b-06a66d10b495"
},
"source": [
"import transformers\n",
"\n",
"print(transformers.__version__)"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"4.11.2\n"
]
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HFASsisvIrIb"
},
"source": [
"You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs [here](https://github.com/huggingface/transformers/tree/master/examples/question-answering)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rEJBSTyZIrIb"
},
"source": [
"# Fine-tuning de un modelo para la tarea de QA"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rFOqCFAwzH-X"
},
"source": [
"En este cuaderno, veremos cómo hacer \"fine-tuning\" a uno de los modelos de [🤗 Transformers](https://github.com/huggingface/transformers) para la tarea de respuesta a una pregunta (QA), que es la tarea de extraer la respuesta a una pregunta de un contexto dado. Veremos cómo cargar fácilmente un conjunto de datos para este tipo de tareas y usar la API `Trainer` para ajustar un modelo en él.\n",
"\n",
"\n",
"\n",
"** Nota: ** Este cuaderno afina los modelos que responden preguntas tomando una subcadena de un contexto, no generando texto nuevo."
]
},
{
"cell_type": "code",
"metadata": {
"id": "zVvslsfMIrIh"
},
"source": [
"squad_v2 = False\n",
"model_checkpoint = \"BSC-TeMU/roberta-base-bne\"\n",
"batch_size = 16"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "whPRbBNbIrIl"
},
"source": [
"## Cargando el dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W7QYTpxXIrIl"
},
"source": [
"Usaremos la biblioteca [🤗 Datasets](https://github.com/huggingface/datasets) para descargar los datos y obtener la métrica que necesitamos usar para la evaluación (para comparar nuestro modelo con el benchmark). Esto se puede hacer fácilmente con las funciones `load_dataset` y` load_metric`."
]
},
{
"cell_type": "code",
"metadata": {
"id": "IreSlFmlIrIm"
},
"source": [
"from datasets import load_dataset, load_metric"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "CKx2zKs5IrIq"
},
"source": [
"Para nuestro ejemplo usaremos el [Dataset SQAC](https://huggingface.co/datasets/BSC-TeMU/SQAC). El cuaderno debe funcionar con cualquier conjunto de datos de respuesta a preguntas proporcionado por la biblioteca 🤗 Datasets. Si está utilizando su propio conjunto de datos definido a partir de un archivo JSON o csv (consulte la [documentación de Datasets[texto del enlace](https://)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) sobre cómo cargarlos ), es posible que necesite algunos ajustes en los nombres de las columnas utilizadas."
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 296,
"referenced_widgets": [
"8da92867c5b44e80ab7a2b4731ede590",
"20a81ea50f5f493182dafc57b0be2c4c",
"6e1e06162c7449b7bcc6a489d665aaaa",
"32a63b55ed1f4cc48d5c4209132929fa",
"317db374e85c49aeb67e7eee7ca9eb9e",
"1031b20b8d2748d386f40f71b091808a",
"a8b23c0c30a54acf99d205c40959dacb",
"fef9bc9e5826482897a0f0a164e4168a",
"43937bb9d5d944c3b030ff863d6aef80",
"f33bb66719f94d479e59a8b8a0ee7106",
"755e9028a4954dfa9a4775336be62040",
"29bfa26690dd43c5918180775b48937e",
"76d523e3477443d794d1c5692ccd9d38",
"c8f9450d7f6e4d90a11e8bcb088c2225",
"200d5ca959054446bc50e2cb4006dbc4",
"720aa191a19044efbddaf1039c45e8a9",
"54b93c05e0d342498020ad8dd0bfc351",
"8877c9db24364603803a7eca08236f05",
"51a17d0911064d54a3a67a492a88ec0e",
"f9b452e4da324032bde0144ae5f0022e",
"fbf1a1f5742d4bb9a5c93fbc64cbc464",
"7ce07165e8294016a7de941738ae8389",
"f1009e1822c14c59ab70039006d0f38e",
"affbb1f7d25d49029b13ae02c73668be",
"8aee3fa61b36413b8d80478c4be09243",
"eb7fe655be5d4aefb2a0c4c26a48796b",
"979ebd3e0db14655b8effb3fbcb336a1",
"a345f336c3dd458a88036aaa9fbc0d1f",
"6082998f17ce4f39b2df2ac3867e500e",
"f6865238c00041aba1b371b0a2c946d5",
"21ccf35dcd024568b8daa0c2a1e2b479",
"cf599c3d13104f44812d8a8b2f1784ce",
"4841318237ce49eab3d28843c335f4ed",
"bf3ef2d0c83f4242982163830f56b0bc",
"e2f7d510f6dc48cba86492836d55a935",
"d5619c4cf98742da8f9938d4a6c0b6d4",
"781fba40d21440458d6bcb3d3072484a",
"188c44a48a16429bb72e816ae164c9b3",
"9503d51d35234c86861d704fb66feb91",
"27076bf4de4045d9b4058921b52f91e0",
"a57abe4411d248269d190f5af32880ca",
"1d2e0967076a42e1a8c8bde745cc35d3",
"16d7bfb41a8341bcbf9cf359fc97a345",
"ae7786adb5e14e8592b1309483280e72",
"0cd9c9f81dde43a7b0011b4609b01a5c",
"416dc21ddac34d5abee1d293ce4f5931",
"a34d5b521df5429bb78ef1cd85e39362",
"c2f718a827674a43a765f772fc67b3d1",
"62240abb9caf423a8acbf0785b656fe9",
"1c263f80958d4bb2aa8141aad2cbe0d5",
"64641b95c0384bf5bf1f8d2a94e2686f",
"dd1e24749ec64c3184698acb25700e98",
"cb2617281cc94fcd98b2288941a64a99",
"62bd5b1222904823a5dd3ba419d1387e",
"66d2d3c701a049e38caf3cbcd19092ba",
"0241437e79d04f81be6968166795d323",
"ece8003b7d7a4741ad111c525ac8b41a",
"9c9052fc37f24b3ca554c8e9e8741e2c",
"30c5046e64e2415f8f12a6b40399fe2c",
"2dfd9d7856a24ea3a0d62839a38c98d8",
"26d9b443f5b94b71b2b111dc3748efdb",
"2d049d96dd5547f0a5885e058fdad375",
"25a6c1634c554967a617e5646ecd81c6",
"46385133fe9f4a1b99ec077db3aec2b6",
"ddf7a78c31ea4faea0b1c7a3e38594b6",
"19a3dac1145c417b8aa0444bd046196c",
"f170c233f06e48239c1b180956ab097e",
"3585870ff79e40faa96a25361c84529b",
"75ec9e40729549c192d4f231e457ed85",
"5fb18fc7ee214a85a2058170100302af",
"71d06d73e5ec4fb29b26c5cdf3060b44",
"0659f1faa05440a8b27be7c07e967ce9",
"2c7deb2f80af42578c4c95f8415ee543",
"a5376d15a8ed41c3aaf60dcae932f147",
"3b76a94d44054e1383e92efd1fe6a0fa",
"7e1dd612a8284dbf8a54e23341971edd",
"52117fc88c7e4972b1051cb7ffcf32e3",
"c80dd3c6d4894a50835263372c65b5bf",
"360178e85d424251823add25e6971275",
"4e58a6f8517e45d1821adcf5ce7ff416",
"0a01025b58a843339777741a0f055088",
"f396653f40394a0aa242e54e88ca47fc",
"ab838d93f88c43c2b92a9f7f2761c67d",
"f2d56bed0410420ca29d707ee5442c8a",
"3d8b544c4d7f4cf2b26ac80e09b9b997",
"03cee110bcfe440bb3f0660ac935f080",
"1a55d593e7a84b6aad88915d24db93d0",
"a031a8a789784f8ca7e7b708fb0e9a7b",
"20774b700c9d4669b4cf7af87b22ce29",
"b97012a8fad34ffc9eae05adb260c018",
"5b35e716d5fa49e0a3f17ceacafa9ef2",
"578f4c280ae945dcb266551cf72df3f4",
"cc978b75fc0d48fbbae205c39afebc65",
"439edf682f8848a5b444900e4d51cb94",
"df8b76c9657b46e794ce224563d81344",
"817687c1baf9464288516376b8ad2e0a",
"f308364939a84955ba7a8ca81c6808d1",
"4e06bfcb009e456b800ca6fa3e0d93e4",
"996f8177c10941c3aee55b57615a8329",
"2f9017a010d54278abf177d2d554e8fa",
"135b344aa24f416099dbadfa6ad9070f",
"3499dd4eb4824588a1cb7a9bca74064f",
"dbac1955c39143bbaf0da37cd2d12384",
"a554026728854d928ff7ebe0a59fce77",
"2eaa764656a44eca9d4a8b533dedc696",
"44acb3acb7444de1a24c20e3702c3c70",
"a03da758ba57490e8ef75067a0c1c11b",
"31f45890df624d58b58f6f5a53504767",
"4eacee5cc37349928d89c36972f3330e",
"c47c066ad5c14218ac55acd190e0a16b"
]
},
"id": "s_AY1ATSIrIq",
"outputId": "1b4d496b-b573-4c61-a2c7-a0c4fdc758d4"
},
"source": [
"datasets = load_dataset(\"BSC-TeMU/SQAC\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8da92867c5b44e80ab7a2b4731ede590",
"version_minor": 0,
"version_major": 2
},
"text/plain": [
"Downloading: 0%| | 0.00/5.56k [00:00\n",
" \n",
" | Epoch | \n", "Training Loss | \n", "Validation Loss | \n", "
|---|---|---|
| 1 | \n", "0.997100 | \n", "0.864572 | \n", "
| 2 | \n", "0.482000 | \n", "0.933443 | \n", "
| 3 | \n", "0.165200 | \n", "1.211125 | \n", "
"
],
"text/plain": [
"