license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- chat
InternLM2.5-7B-Chat GGUF Model
Introduction
The internlm2_5-7b-chat model in GGUF format can be utilized by llama.cpp, a highly popular open-source framework for Large Language Model (LLM) inference, across a variety of hardware platforms, both locally and in the cloud.
This repository offers internlm2_5-7b-chat models in GGUF format in both half precision and various low-bit quantized versions, including q5_0, q5_k_m, q6_k, and q8_0.
In the subsequent sections, we will first present the installation procedure, followed by an explanation of the model download process. And finally we will illustrate the methods for model inference and service deployment through specific examples.
Installation
We recommend building llama.cpp from source. The following code snippet provides an example for the Linux CUDA platform. For instructions on other platforms, please refer to the official guide.
- Step 1: create a conda environment and install cmake
conda create --name internlm2 python=3.10 -y
conda activate internlm2
pip install cmake
- Step 2: clone the source code and build the project
git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
All the built targets can be found in the sub directory build/bin
In the following sections, we assume that the working directory is at the root directory of llama.cpp.
Download models
In the introduction section, we mentioned that this repository includes several models with varying levels of computational precision. You can download the appropriate model based on your requirements.
For instance, internlm2_5-7b-chat-fp16.gguf can be downloaded as below：
pip install huggingface-hub
huggingface-cli download internlm/internlm2_5-7b-chat-gguf internlm2_5-7b-chat-fp16.gguf --local-dir . --local-dir-use-symlinks False
Inference
You can use llama-cli for conducting inference. For a detailed explanation of llama-cli, please refer to this guide
build/bin/llama-cli -m internlm2_5-7b-chat-fp16.gguf -ngl 32
Serving
llama.cpp provides an OpenAI API compatible server - llama-server. You can deploy internlm2_5-7b-chat-fp16.gguf into a service like this:
./build/bin/llama-server -m ./internlm2_5-7b-chat-fp16.gguf -ngl 32
At the client side, you can access the service through OpenAI API:
from openai import OpenAI
client = OpenAI(
api_key='YOUR_API_KEY',
base_url='http://localhost:8080/v1'
)
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
model=model_name,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": " provide three suggestions about time management"},
],
temperature=0.8,
top_p=0.8
)
print(response)