Update README.md

README.md

@@ -116,7 +116,71 @@ The AutoAWQ script has been adapted from [AutoAWQ/examples/generate.py](https://
 
 ### 🤗 Text Generation Inference (TGI)
 
-Coming soon!
+To run the `text-generation-launcher` with Llama 3.1 70B Instruct AWQ in INT4 with Marlin kernels for optimized inference speed, you will need to have Docker installed (see [installation notes](https://docs.docker.com/engine/install/)) and the `huggingface_hub` Python package as you need to login to the Hugging Face Hub.
+
+```bash
+pip install -q --upgrade huggingface_hub
+huggingface-cli login
+```
+
+Then you just need to run the TGI v2.2.0 (or higher) Docker container as follows:
+
+```bash
+docker run --gpus all --shm-size 1g -ti -p 8080:80 \
+    -e MODEL_ID=hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
+    -e NUM_SHARD=4 \
+    -e QUANTIZE=awq \
+    -e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
+    -e MAX_INPUT_LENGTH=4000 \
+    -e MAX_TOTAL_TOKENS=4096 \
+    ghcr.io/huggingface/text-generation-inference:2.2.0
+```
+
+> [!NOTE]
+> TGI will expose different endpoints, to see all the endpoints available check [TGI OpenAPI Specification](https://huggingface.github.io/text-generation-inference/#/).
+
+To send request to the deployed TGI endpoint compatible with [OpenAI specification](https://github.com/openai/openai-openapi) i.e. `/v1/chat/completions`:
+
+```bash
+curl 0.0.0.0:8080/v1/chat/completions \
+    -X POST \
+    -H 'Content-Type: application/json' \
+    -d '{
+        "model": "tgi",
+        "messages": [
+            {
+                "role": "system",
+                "content": "You are a helpful assistant."
+            },
+            {
+                "role": "user",
+                "content": "What is Deep Learning?"
+            }
+        ],
+        "max_tokens": 128
+    }'
+```
+
+Or via the `openai` Python SDK (see [installation notes](https://github.com/openai/openai-python?tab=readme-ov-file#installation)) as:
+
+```python
+import os
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://0.0.0.0:8080/v1/",
+    api_key=os.getenv("HF_TOKEN"),
+)
+
+chat_completion = client.chat.completions.create(
+    model="tgi",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "What is Deep Learning?"},
+    ],
+    max_tokens=128,
+)
+```
 
 ## Quantization Reproduction