File size: 13,899 Bytes
59699eb
34fd201
 
 
 
 
 
 
 
 
59699eb
885f9cf
 
 
 
 
 
 
 
 
 
 
59699eb
 
 
a020ad4
59699eb
77048f9
 
e1256fa
02050fe
e1256fa
77048f9
 
59699eb
 
77048f9
59699eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cfbc085
 
59699eb
 
 
 
 
cfbc085
 
59699eb
 
 
77048f9
59699eb
 
 
 
 
 
 
 
77048f9
 
59699eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e1256fa
59699eb
 
 
 
 
 
77048f9
59699eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77048f9
59699eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77048f9
59699eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77048f9
59699eb
 
 
 
 
 
 
 
 
 
 
77048f9
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
---
extra_gated_heading: >-
  Acknowledge to follow corresponding license to access the
  repository
extra_gated_button_content: Agree and access repository
extra_gated_fields:
  First Name: text
  Last Name: text
  Country: country
  Affiliation: text
license: cc-by-nc-4.0
datasets:
- Salesforce/xlam-function-calling-60k
language:
- en
pipeline_tag: text-generation
tags:
- function-calling
- LLM Agent
- tool-use
- deepseek
- pytorch
---

<p align="center">
<img width="500px" alt="xLAM" src="https://huggingface.co/datasets/jianguozhang/logos/resolve/main/xlam-no-background.png">
</p>
<p align="center">
  <a href="https://apigen-pipeline.github.io/">[Homepage]</a>  |  
  <a href="https://arxiv.org/abs/2406.18518">[Paper]</a> | 
  <a href="https://discord.gg/tysWwgZyQ2">[Discord]</a> | 
  <a href="https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k">[Dataset]</a> | 
  <a href="https://github.com/SalesforceAIResearch/xLAM">[Github]</a>
</p>
<hr>


Welcome to the xLAM model family! [Large Action Models (LAMs)](https://blog.salesforceairesearch.com/large-action-models/) are advanced large language models designed to enhance decision-making and translate user intentions into executable actions that interact with the world. LAMs autonomously plan and execute tasks to achieve specific goals, serving as the brains of AI agents. They have the potential to automate workflow processes across various domains, making them invaluable for a wide range of applications.

## Table of Contents
- [Model Series](#model-series)
- [Repository Overview](#repository-overview)
- [Benchmark Results](#benchmark-results)
- [Usage](#usage)
  - [Basic Usage with Huggingface](#basic-usage-with-huggingface)
  - [Usage with vLLM](#usage-with-vllm)
- [License](#license)
- [Citation](#citation)

## Model Series

We provide a series of xLAMs in different sizes to cater to various applications, including those optimized for function-calling and general agent applications:

| Model                  | # Total Params | Context Length | Download Model  | Download GGUF files |
|------------------------|----------------|----------------|----------------|----------|
| xLAM-1b-fc-r           | 1.35B          | 16384            | [🤗 Link](https://huggingface.co/Salesforce/xLAM-1b-fc-r) | [🤗 Link](https://huggingface.co/Salesforce/xLAM-1b-fc-r-gguf) |
| xLAM-7b-fc-r           | 6.91B          | 4096             | [🤗 Link](https://huggingface.co/Salesforce/xLAM-7b-fc-r) | [🤗 Link](https://huggingface.co/Salesforce/xLAM-7b-fc-r-gguf) |

The `fc` series of models are optimized for function-calling capability, providing fast, accurate, and structured responses based on input queries and available APIs. These models are fine-tuned based on the [deepseek-coder](https://huggingface.co/collections/deepseek-ai/deepseek-coder-65f295d7d8a0a29fe39b4ec4) models and are designed to be small enough for deployment on personal devices like phones or computers.

We also provide their quantized [GGUF](https://huggingface.co/docs/hub/en/gguf) files for efficient deployment and execution. GGUF is a file format designed to efficiently store and load large language models, making GGUF ideal for running AI models on local devices with limited resources, enabling offline functionality and enhanced privacy.

For more details, check our [GitHub](https://github.com/SalesforceAIResearch/xLAM) and [paper](https://arxiv.org/abs/2406.18518).


## Repository Overview

This repository is focused on our small `xLAM-7b-fc-r` model, which is optimized for function-calling and can be easily deployed on personal devices.

<div align="center">
<img src="https://github.com/apigen-pipeline/apigen-pipeline.github.io/blob/main/img/function-call-overview.png?raw=true"
alt="drawing" width="620"/>
</div>

Function-calling, or tool use, is one of the key capabilities for AI agents. It requires the model not only understand and generate human-like text but also to execute functional API calls based on natural language instructions. This extends the utility of LLMs beyond simple conversation tasks to dynamic interactions with a variety of digital services and applications, such as retrieving weather information, managing social media platforms, and handling financial services.

The instructions will guide you through the setup, usage, and integration of `xLAM-7b-fc-r` with HuggingFace and vLLM.
We will first introduce the basic usage, and then walk through the provided tutorial and example scripts in the [examples](https://huggingface.co/Salesforce/xLAM-7b-fc-r/tree/main/examples) folder.

### Framework Versions

- Transformers 4.41.0
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1

## Benchmark Results

We mainly test our function-calling models on the [Berkeley Function-Calling Leaderboard (BFCL)](https://gorilla.cs.berkeley.edu/leaderboard.html), which offers a comprehensive evaluation framework for assessing LLMs' function-calling capabilities across various programming languages and application domains like Java, JavaScript, and Python. 


<div align="center">
<img src="https://github.com/apigen-pipeline/apigen-pipeline.github.io/blob/main/img/table-result-0718.png?raw=true" width="620" alt="Performance comparison on Berkeley Function-Calling Leaderboard">
<p>Performance comparison on the BFCL benchmark as of date 07/18/2024. Evaluated with <code>temperature=0.001</code> and <code>top_p=1</code></p> 
</div>

<p>Our <code>xLAM-7b-fc-r</code> secures the 3rd place with an overall accuracy of 88.24% on the leaderboard, outperforming many strong models. Notably, our <code>xLAM-1b-fc-r</code> model is the only tiny model with less than 2B parameters on the leaderboard, but still achieves a competitive overall accuracy of 78.94% and outperforming GPT3-Turbo and many larger models.
Both models exhibit balanced performance across various categories, showing their strong function-calling capabilities despite their small sizes.</p>

See our [paper](https://arxiv.org/abs/2406.18518) and Github [repo](https://github.com/SalesforceAIResearch/xLAM) for more detailed analysis.


## Usage

### Basic Usage with Huggingface

To use the `xLAM-7b-fc-r` model from Huggingface, please first install the `transformers` library:
```bash
pip install transformers>=4.41.0
```

We use the following example to illustrate how to use our model to perform function-calling tasks. 
Please note that, our model works best with our provided prompt format. 
It allows us to extract JSON output that is similar to the [function-calling mode of ChatGPT](https://platform.openai.com/docs/guides/function-calling).

````python
import json
import torch 
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.random.manual_seed(0) 

model_name = "Salesforce/xLAM-7b-fc-r"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name) 

# Please use our provided instruction prompt for best performance
task_instruction = """
You are an expert in composing functions. You are given a question and a set of possible functions. 
Based on the question, you will need to make one or more function/tool calls to achieve the purpose. 
If none of the functions can be used, point it out and refuse to answer. 
If the given question lacks the parameters required by the function, also point it out.
""".strip()

format_instruction = """
The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
```
{
    "tool_calls": [
    {"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},
    ... (more tool calls as required)
    ]
}
```
""".strip()

# Define the input query and available tools
query = "What's the weather like in New York in fahrenheit?"

get_weather_api = {
    "name": "get_weather",
    "description": "Get the current weather for a location",
    "parameters": {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, New York"
            },
            "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "The unit of temperature to return"
            }
        },
        "required": ["location"]
    }
}

search_api = {
    "name": "search",
    "description": "Search for information on the internet",
    "parameters": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "The search query, e.g. 'latest news on AI'"
            }
        },
        "required": ["query"]
    }
}

openai_format_tools = [get_weather_api, search_api]

# Helper function to convert openai format tools to our more concise xLAM format
def convert_to_xlam_tool(tools):
    ''''''
    if isinstance(tools, dict):
        return {
            "name": tools["name"],
            "description": tools["description"],
            "parameters": {k: v for k, v in tools["parameters"].get("properties", {}).items()}
        }
    elif isinstance(tools, list):
        return [convert_to_xlam_tool(tool) for tool in tools]
    else:
        return tools

# Helper function to build the input prompt for our model
def build_prompt(task_instruction: str, format_instruction: str, tools: list, query: str):
    prompt = f"[BEGIN OF TASK INSTRUCTION]\n{task_instruction}\n[END OF TASK INSTRUCTION]\n\n"
    prompt += f"[BEGIN OF AVAILABLE TOOLS]\n{json.dumps(xlam_format_tools)}\n[END OF AVAILABLE TOOLS]\n\n"
    prompt += f"[BEGIN OF FORMAT INSTRUCTION]\n{format_instruction}\n[END OF FORMAT INSTRUCTION]\n\n"
    prompt += f"[BEGIN OF QUERY]\n{query}\n[END OF QUERY]\n\n"
    return prompt
    
# Build the input and start the inference
xlam_format_tools = convert_to_xlam_tool(openai_format_tools)
content = build_prompt(task_instruction, format_instruction, xlam_format_tools, query)

messages=[
    { 'role': 'user', 'content': content}
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

# tokenizer.eos_token_id is the id of <|EOT|> token
outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))
````

Then you should be able to see the following output string in JSON format:

```shell
{"tool_calls": [{"name": "get_weather", "arguments": {"location": "New York", "unit": "fahrenheit"}}]}
```

We highly recommend to use our provided prompt format and helper functions to yield the best function-calling performance of our model.

### Usage with vLLM

We provide example scripts to deploy our model with `vllm` and run inferences. First, install the required packages:

```bash
pip install vllm openai argparse jinja2
```

The example scripts are located in the [examples](https://huggingface.co/Salesforce/xLAM-7b-fc-r/tree/main/examples) folder.

#### 1. Test Prompt Template

To build prompts using the chat template and output formatted prompts ready for various test cases, run:

```bash
python test_prompt_template.py --model 
```

#### 2. Test xLAM Model with a Manually Served Endpoint

a. Serve the model with vLLM:

```bash
python -m vllm.entrypoints.openai.api_server --model Salesforce/xLAM-7b-fc-r --served-model-name xLAM-7b-fc-r --dtype bfloat16 --port 8001
```

b. Run the test script:

```bash
python test_xlam_model_with_endpoint.py --model_name xLAM-7b-fc-r --port 8001 [OPTIONS]
```

Options:
- `--temperature`: Default 0.3
- `--top_p`: Default 1.0
- `--max_tokens`: Default 512

This test script provides a handler implementation that can be easily applied to your customized function-calling applications.

#### 3. Test xLAM Model by Directly Using vLLM Library

To test the xLAM model directly with the vLLM library, run:

```bash
python test_xlam_model_with_vllm.py --model Salesforce/xLAM-7b-fc-r [OPTIONS]
```

Options are the same as for the endpoint test. This test script also provides a handler implementation that can be easily applied to your customized function-calling applications.

#### Customization

These examples are designed to be flexible and easily integrated into your own projects. Feel free to modify the scripts to suit your specific needs and applications. You can adjust test queries or API definitions in each script to test different scenarios or model capabilities.

Additional customization tips:
- Modify the `--dtype` parameter when serving the model based on your GPU capacity.
- Refer to the vLLM documentation for more detailed configuration options.
- Explore the `demo.ipynb` file for a comprehensive description of the entire workflow, including how to execute APIs.

These resources provide a robust foundation for integrating xLAM models into your applications, allowing for tailored and efficient deployment.

## License

`xLAM-7b-fc-r` is distributed under the CC-BY-NC-4.0 license, with additional terms specified in the [Deepseek license](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL).

## Citation

If you find this repo helpful, please cite our paper:
```bibtex
@article{liu2024apigen,
  title={APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author={Liu, Zuxin and Hoang, Thai and Zhang, Jianguo and Zhu, Ming and Lan, Tian and Kokane, Shirley and Tan, Juntao and Yao, Weiran and Liu, Zhiwei and Feng, Yihao and others},
  journal={arXiv preprint arXiv:2406.18518},
  year={2024}
}
```