|
--- |
|
pipeline_tag: text-generation |
|
inference: true |
|
widget: |
|
- text: 'def print_hello_world():' |
|
example_title: Hello world |
|
group: Python |
|
license: bigscience-openrail-m |
|
pretrain-datasets: |
|
- books |
|
- arxiv |
|
- c4 |
|
- falcon-refinedweb |
|
- wiki |
|
- github-issues |
|
- stack_markdown |
|
- self-made dataset of permissive github code |
|
datasets: |
|
- bigcode/the-stack-dedup |
|
- rombodawg/2XUNCENSORED_MegaCodeTraining188k |
|
- bigcode/commitpackft |
|
library_name: llama.cpp |
|
tags: |
|
- code |
|
language: |
|
- en |
|
--- |
|
|
|
# Refact 1.6B FIM GGUF |
|
|
|
## Introduction |
|
|
|
The Refact 1.6B FIM GGUF model is a state-of-the-art AI-powered coding assistant developed by Small Magellanic Cloud AI Ltd. This model is designed to assist developers with code completion, refactoring, and chat-based interactions, excelling in code-related natural language understanding and generation tasks. |
|
|
|
## Quantized Model Files |
|
|
|
The model comes in various quantized versions to suit different computational needs: |
|
|
|
- **refact-1.6B-fim-q4_0.gguf**: A 4-bit quantized model with a file size of 878 MB. |
|
- **refact-1.6B-fim-q5_0.gguf**: A 5-bit quantized model with a file size of 1.1 GB. |
|
- **refact-1.6B-fim-q8_0.gguf**: An 8-bit quantized model with a file size of 1.6 GB. |
|
|
|
## Features and Usage |
|
|
|
The model is versatile and can be employed for: |
|
|
|
- Code completion |
|
- Code refactoring |
|
- Chat-based interactions |
|
|
|
### Example Usage |
|
|
|
Here's a sample shell command to invoke the model: |
|
|
|
```sh |
|
# Sample shell command to use the model |
|
./main -m models/smallcloudai/Refact-1_6B-fim/ggml-model-f16.gguf -n 300 -p "write a function to multiply two integers in python" --temp 1.0 --top-p 1.0 --top-k 1 --repeat_penalty 1.0 |
|
``` |
|
|
|
## Performance Metrics |
|
|
|
The model outperforms many existing models in both code completion and chat-based interactions, as evidenced by the HumanEval results. |
|
|
|
| Model | Size | HumanEval pass@1 | HumanEval pass@10 | |
|
|----------------------|-------|------------------|-------------------| |
|
| **Refact-1.6-fim** | 1.6b | 32.0% | 53.0% | |
|
| StableCode | 3b | 20.2% | 33.8% | |
|
| ReplitCode v1 | 3b | 21.9% | N/A | |
|
|
|
## Installation and Setup |
|
|
|
The model can be integrated into your IDE via the [Refact plugin](https://refact.ai/). For self-hosting, an [open-source Docker container](https://github.com/smallcloudai/refact) is available. |
|
|
|
## Limitations and Bias |
|
|
|
The model primarily focuses on English text, which may result in lower performance for non-English languages. |
|
|
|
## Technical Specifications |
|
|
|
- **Architecture**: LLAMA-like model with multi-query attention |
|
- **Training Tokens**: 1.2T for pretraining, 40B for fine-tuning |
|
- **Precision**: bfloat16 |
|
- **Training Time**: 28 days |
|
|
|
## License |
|
|
|
The model is licensed under the BigScience OpenRAIL-M v1 license agreement. |
|
|
|
## Citation |
|
|
|
If you use this model in your work, please cite it by linking back to the following page for proper attribution: |
|
|
|
[Refact 1.6B FIM Model](https://huggingface.co/smallcloudai/Refact-1_6B-fim) |
|
|
|
## Acknowledgments |
|
|
|
Special thanks to [ds5t5](https://github.com/ggerganov/llama.cpp/pull/3329) for their contribution in implementing the source for converting the model's tensors from Hugging Face to GGUF format. Their work has been instrumental in enhancing the model's versatility. |
|
|
|
### Example Command for Testing |
|
|
|
To test the model against Hugging Face, you can use the following command: |
|
|
|
```sh |
|
# Example command for testing against Hugging Face |
|
python3 convert-refact-hf-to-gguf.py ./Refact-1_6B-fim 1 |
|
|
|
./main -m ./Refact-1_6B-fim/ggml-model-f16.gguf -n 300 -p "write a function to multiply two integers in python" --temp 1.0 --top-p 1.0 --top-k 1 --repeat_penalty 1.0 |
|
``` |
|
|
|
This resolves llama.cpp issue [#3061](https://github.com/ggerganov/llama.cpp/issues/3061). |
|
|
