Linking to the blog

notebooks/jais_tgi_inference_endpoints.ipynb

@@ -2,69 +2,11 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "5d9aca72-957a-4ee2-862f-e011b9cd3a62",
+   "id": "db41d8ba-71c0-4951-9a88-e1ae01a282ec",
    "metadata": {},
    "source": [
     "# Introduction\n",
-    "## Goal\n",
-    "I want [jais-13B](https://huggingface.co/core42/jais-13b-chat) deployed with an API quickly and easily. I'm also scared of mice so ideally I can just use my keyboard. \n",
-    "\n",
-    "## Approach\n",
-    "There are lots of options out there that are \"1-click\" which is really cool! I would like to do even better and make a \"0-click\". This is great for those that are musophobic (scared of mice) or want scripts that can run without human intervention.\n",
-    "\n",
-    "We will be using [Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) as our serving toolkit as it is robust and configurable. For our hardware we will be using [Inference Endpoints](https://huggingface.co/inference-endpoints) as it makes the deployment procedure really easy! We will be using the API to reach our aforementioned \"0-click\" goal."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2086a136-6710-45af-b2b1-7224b5cbbca7",
-   "metadata": {},
-   "source": [
-    "# Pre-requisites\n",
-    "Deploying LLMs is a tough process. There are a number of challenges! \n",
-    "- These models are huge\n",
-    "    - Slow to load \n",
-    "    - Won't fit on convenient HW\n",
-    "- Generative transformers require iterative decoding\n",
-    "- Many of the optimizations are not consolidated\n",
-    "\n",
-    "TGI solves many of these, and while I don't want to dedicate this blog to TGI there are a few concepts we need to cover to properly understand how to configure our deployment.\n",
-    "\n",
-    "\n",
-    "## Prefilling Phase\n",
-    "> In the prefill phase, the LLM processes the input tokens to compute the intermediate states (keys and values), which are used to generate the “first” new token. Each new token depends on all the previous tokens, but because the full extent of the input is known, at a high level this is a matrix-matrix operation that’s highly parallelized. It effectively saturates GPU utilization.\n",
-    "\n",
-    "~[Nvidia Blog](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)\n",
-    "\n",
-    "Prefilling is relatively fast.\n",
-    "\n",
-    "## Decoding Phase\n",
-    "> In the decode phase, the LLM generates output tokens autoregressively one at a time, until a stopping criteria is met. Each sequential output token needs to know all the previous iterations’ output states (keys and values). This is like a matrix-vector operation that underutilizes the GPU compute ability compared to the prefill phase. The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation.\n",
-    "\n",
-    "~[Nvidia Blog](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)\n",
-    "\n",
-    "Decoding is relatively slow.\n",
-    "\n",
-    "## Example\n",
-    "Lets take an example of sentiment analysis:\n",
-    "\n",
-    "Below we have input tokens that the LLM will pre-fill. Note that we know what the next token is during the pre-filling phase. We can use this to our advantage.\n",
-    "```text\n",
-    "### Instruction: What is the sentiment of the input?\n",
-    "### Examples\n",
-    "I wish the screen was bigger - Negative\n",
-    "I hate the battery - Negative\n",
-    "I love the default appliations - Positive\n",
-    "### Input\n",
-    "I am happy with this purchase - \n",
-    "### Response\n",
-    "```\n",
-    "\n",
-    "Below we have output tokens generated during decoding phase. Despite being few in this example we dont know what the next token will be until we have generated it.\n",
-    "\n",
-    "```text\n",
-    "Positive\n",
-    "```"
+    "Please check out my [blog post](https://datavistics.github.io/posts/jais-inference-endpoints/) for more details!"
    ]
   },
   {
@@ -142,7 +84,7 @@
    "metadata": {},
    "source": [
     "## Config\n",
-    "You need to fill this in with your desired repos. Note I used 5 for the `MAX_WORKERS` since `jina-embeddings-v2` are quite memory hungry. "
+    "Choose your `ENDPOINT_NAME` if you like."
    ]
   },
   {
@@ -191,7 +133,7 @@
    "source": [
     "Some users might have payment registered in an organization. This allows you to connect to an organization (that you are a member of) with a payment method.\n",
     "\n",
-    "Leave it blank is you want to use your username."
+    "Leave it blank if you want to use your username."
    ]
   },
   {
@@ -267,32 +209,6 @@
     ")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "bbc82ce5-d7fa-4167-adc1-b25e567f5559",
-   "metadata": {},
-   "source": [
-    "This is one of the most important parts of this tutorial to understand well. Its important that we choose the deployment settings that best represent our needs and our hardware. I'll just leave some high-level information here and we can go deeper in a future tutorial. It would be interesting to show the difference in how you would optimize your deployment between a chat application and RAG.\n",
-    "\n",
-    "`MAX_BATCH_PREFILL_TOKENS` | [docs](https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#maxbatchprefilltokens) |\n",
-    "> Limits the number of tokens for the prefill operation. Since this operation take the most memory and is compute bound, it is interesting to limit the number of requests that can be sent\n",
-    "\n",
-    "`MAX_INPUT_LENGTH` | [docs](https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#maxinputlength) |\n",
-    "> This is the maximum allowed input length (expressed in number of tokens) for users. The larger this value, the longer prompt users can send which can impact the overall memory required to handle the load. Please note that some models have a finite range of sequence they can handle\n",
-    "\n",
-    "I left this quite large as I want to give a lot of freedom to the user more than I want to trade performance. It's important in RAG applications to give more freedom here. But for few turn chat applications you can be more restrictive.\n",
-    "\n",
-    "`MAX_TOTAL_TOKENS` | [docs](https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#maxtotaltokens) | \n",
-    "> This is the most important value to set as it defines the \"memory budget\" of running clients requests. Clients will send input sequences and ask to generate `max_new_tokens` on top. with a value of `1512` users can send either a prompt of `1000` and ask for `512` new tokens, or send a prompt of `1` and ask for `1511` max_new_tokens. The larger this value, the larger amount each request will be in your RAM and the less effective batching can be.\n",
-    "\n",
-    "`TRUST_REMOTE_CODE` This is set to `true` as jais requires it.\n",
-    "\n",
-    "`QUANTIZE` | [docs](https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#quantize) |\n",
-    "> Whether you want the model to be quantized\n",
-    "\n",
-    "With jais, you really only have the bitsandbytes option. The tradeoff is that inference is a bit slower, but you can use much smaller GPUs (~3x smaller) without noticably losing performance. It's one of the better reads IMO and I recommend checking out the [paper](https://arxiv.org/abs/2208.07339)."
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 7,
@@ -461,10 +377,10 @@
    "id": "41abea64-379d-49de-8d9a-355c2f4ce1ac",
    "metadata": {},
    "source": [
-    "# Analyze Usage\n",
+    "## Analyze Usage\n",
     "1. Go to your `dashboard_url` printed below\n",
-    "1. Click on the Usage & Cost tab\n",
-    "1. See how much you have spent"
+    "1. Check the dashboard\n",
+    "1. Analyze the Usage & Cost tab"
    ]
   },
   {
@@ -493,8 +409,7 @@
    "id": "b953d5be-2494-4ff8-be42-9daf00c99c41",
    "metadata": {},
    "source": [
-    "# Delete Endpoint\n",
-    "We should see a `200` if everything went correctly."
+    "## Delete Endpoint"
    ]
   },
   {