commit message

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+models/glm6b-kv-cache-dy-bs8.ftm filter=lfs diff=lfs merge=lfs -text
+models/glm6b-bs8.ftm filter=lfs diff=lfs merge=lfs -text
+*.so filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,6 @@
+dist/
+*.egg-info/
+__pycache__
+build/
+.vscode
+.idea

README.md

@@ -0,0 +1,123 @@
+---
+license: mit
+language: en
+tags:
+- LLM
+- LLaMA-13b
+---
+## Special Notes
+
+Due to the license regulation of LLaMA, we are not allowed to release the accelerated parameters directly. We hope to discuss with you guys to figure out **a legal way** to share lyraLLaMA. If you any any suggestions, please feel free to drop us a line at [email protected].
+
+## Model Card for lyraLLaMA
+
+lyraLLaMA is currently the **fastest LLaMA-13b** available. The inference speed of lyraLLaMA has achieved **4x** acceleration upon the torch version.
+
+Among its main features are:
+- device: Nvidia GPU with Amperer architecture or Volta architecture (A100 or higher, V100).
+- batch_size: compiled with dynamic batch size, maximum depends on device. 
+
+We use the LLaMA.13B model for measurement, but this optimized inference is applicable to LLaMA models of different sizes.
+
+## Speed
+
+* test on A100 40G
+* fp16 precision
+
+### LLaMA-Ziya-13B
+
+| Version | Batch Size 1 | Batch Size 8 | Batch Size 16 | Batch Size 32 | Batch Size 64 |
+| --- | --- | --- | --- | --- | --- |
+| Torch LLaMA | 31.74 | 289.2 | 521.37 | 775.69 | OOM |
+| lyraLLaMA | 73.2 | 565.6 | 1179.59 | 1795.63 | 3061.27 |
+
+### LLaMA-Vicuna-13B
+
+| Version | Batch Size 1 | Batch Size 8 | Batch Size 16 | Batch Size 32 | Batch Size 64 |
+| --- | --- | --- | --- | --- | --- |
+| Torch LLaMA | 24.65| 167.3 | 322.97 | 407.99 | OOM |
+| lyraLLaMA | 53.67 | 421.38 | 804.31 | 151.928| 2679.82 |
+
+## Docker Environment Recommendation
+
+- For Cuda 11.X: we recommend ```nvcr.io/nvidia/pytorch:22.12-py3```
+- For Cuda 12.0: we recommend ```nvcr.io/nvidia/pytorch:23.02-py3```
+
+```bash
+docker pull nvcr.io/nvidia/pytorch:23.02-py3
+docker run --rm -it --gpus all -v ./:/lyraLLaMA nvcr.io/nvidia/pytorch:23.02-py3
+
+pip install -r requirements.txt
+python demo.py
+```
+
+## Uses
+
+```python
+from lyra_llama import lyraLLaMA
+
+model_path = "./models/lamma-13b-1-gpu-fp16.bin"
+tokenizer_path = "./models/"
+dtype='fp16'
+prompt = "列出3个不同的机器学习算法，并说明它们的适用范围"
+max_output_length = 512
+
+model = lyraLLaMA(model_path, tokenizer_path, dtype)
+
+prompt = '<human>:' + prompt.strip() + '\n<bot>:'
+
+bs = 1
+prompts = [prompt, ] * bs
+output_texts = model.generate(
+        prompts, output_length=max_output_length,
+        top_k=30, top_p=0.85, temperature=1.0, repetition_penalty=1.0, do_sample=False)
+
+print(output_texts)
+
+```
+
+## Demo output
+
+### LLaMA-Ziya-13B
+#### input
+列出3个不同的机器学习算法，并说明它们的适用范围.
+
+#### output
+以下是三个常见的机器学习算法及其适用范围:
+
+1. 决策树(Decision Tree):决策树是一种基于分类和回归问题的朴素贝叶斯模型。它通过构建一系列逐步分裂的分支来预测结果。适用于那些具有简单特征、大量数据且数据集大小在可接受范围内的情况。
+
+2. 随机森林(Random Forest):随机森林是一种集成学习算法,由多个决策树组成。它的优点是能够处理大规模数据和高维度的特征。适用于需要对多个变量进行建模的场景,例如医疗诊断、金融风险评估等。
+
+3. 支持向量机(Support Vector Machine):支持向量机是一种监督学习方法,通常用于分类问题。它可以处理高维数据,并且具有较高的准确性。适用于需要对高维数据进行分类或回归的问题,例如图像识别、自然语言处理等。
+
+### LLaMA-Vicuna-13B
+
+#### input
+List 3 different machine learning algorithms and explain where they are applicable.
+
+#### output
+
+Outputs: 
+1. Linear Regression: Linear regression is a simple, yet powerful algorithm that is used to predict the value of a continuous variable based on one or more input features. It is commonly used in prediction and forecasting tasks, such as predicting the price of a stock or the sales of a product.
+2. Decision Trees: Decision Trees are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by partitioning the feature space into smaller subspaces, with each subspace corresponding to a leaf node in the tree. Decision Trees are commonly used in applications such as credit risk assessment and customer segmentation.
+3. Support Vector Machines (SVMs): SVMs are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by finding the best hyperplane that separates the data into different classes. SVMs are commonly used in applications such as image classification and natural language processing.
+
+## TODO
+1. Support for int8 and int4
+2. Inference for longer context situations
+3. Streaming inference mode.
+
+## Citation
+``` bibtex
+@Misc{lyraLLaMA2023,
+  author =       {Kangjian Wu, Zhengtao Wang, Yibo Lu, Bin Wu},
+  title =        {lyraLLaMA: Accelerating LLaMA-13b(fp16) to 3000+ tokens/s},
+  howpublished = {\url{https://huggingface.co/TMElyralab/lyraLLaMA}},
+  year =         {2023}
+}
+```
+
+## Report bug
+- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraLLaMA/discussions
+- report bug with a `[bug]` mark in the title.

demo.py

@@ -0,0 +1,19 @@
+from lyra_llama import lyraLLaMA
+
+model_path = "./models/lamma-13b-1-gpu-fp16.bin"
+tokenizer_path = "./models/"
+dtype='fp16'
+prompt = "列出3个不同的机器学习算法，并说明它们的适用范围"
+max_output_length = 512
+
+model = lyraLLaMA(model_path, tokenizer_path, dtype)
+
+prompt = '<human>:' + prompt.strip() + '\n<bot>:'
+
+bs = 1
+prompts = [prompt, ] * bs
+output_texts = model.generate(
+        prompts, output_length=max_output_length,
+        top_k=30, top_p=0.85, temperature=1.0, repetition_penalty=1.0, do_sample=False)
+
+print(output_texts)

lyra_llama/__init__.py

@@ -0,0 +1 @@
+from .lyra_llama import lyraLlama

lyra_llama/config.py

@@ -0,0 +1,33 @@
+import dataclasses
+from typing import Optional
+
+@dataclasses.dataclass
+class LyraLLaMAParam:
+    num_heads: int = 40
+    size_per_head: int = 128
+    inter_size: int = 13824
+    num_layers: int = 40
+    vocab_size: int = 39424
+    start_id: Optional[int] = 1
+    end_id: Optional[int] = 2
+    tensor_para_size: int = 1
+    pipeline_para_size: int = 1
+    remove_padding: bool = True
+    shared_contexts_ratio: float = 1.0
+    layernorm_eps: float = 1e-6
+    weights_data_type: str = "fp16"
+    rotary_embedding: int = 128
+    use_gptj_residual: bool = False
+
+    def __post_init__(self):
+        if not 0.0 <= self.shared_contexts_ratio <= 1.0:
+            raise ValueError(
+                f'Got an invalid value of shared_context_ratio '
+                f'{self.shared_contexts_ratio} - range: [0.0, 1.0]')
+
+    def asdict(self):
+        return dataclasses.asdict(self)
+
+
+LYRA_LLAMA_PARAM = LyraLLaMAParam()
+LIB_SO_PATH = '/app/LyraLLaMAPy/ftlib/libth_transformer_sm80_cu11.so'

lyra_llama/lyra_llama.py

@@ -0,0 +1,169 @@
+from __future__ import annotations
+
+import configparser
+import pathlib
+import typing
+
+import torch
+import transformers
+from torch.nn.utils.rnn import pad_sequence
+
+from .config import LYRA_LLAMA_PARAM, LIB_SO_PATH
+from .model import LLaMAModel
+
+class lyraLLaMA:
+    def __init__(self, model_path, tokenizer_path=None, dtype='fp16', int8_mode=0) -> None:
+        self.model_path = model_path
+        self.tokenizer_path = tokenizer_path
+        self.dtype = dtype
+        if dtype != 'int8':
+            int8_mode = 0
+        self.int8_mode = int8_mode
+
+        self.model, self.tokenizer = self.load_model_and_tokenizer()
+        print("Got model and tokenizer")
+
+    def load_model_and_tokenizer(self):
+        if self.tokenizer_path is None:
+            tokenizer_path = self.model_path
+        else:
+            tokenizer_path = self.tokenizer_path
+
+        print(f'Loading tokenizer from {tokenizer_path}')
+        tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_path)
+
+        checkpoint_path = pathlib.Path(self.tokenizer_path)
+        config_path = checkpoint_path / 'config.ini'
+
+        if config_path.exists():
+            # Read model params from config.
+            cfg = configparser.ConfigParser()
+            cfg.read(config_path)
+            model_name = 'llama'
+            inference_data_type = self.dtype
+            if inference_data_type == None:
+                inference_data_type = cfg.get(model_name, "weight_data_type")
+            model_args = dict(
+                head_num=cfg.getint(model_name, 'head_num'),
+                size_per_head=cfg.getint(model_name, "size_per_head"),
+                layer_num=cfg.getint(model_name, "num_layer"),
+                tensor_para_size=cfg.getint(model_name, "tensor_para_size"),
+                vocab_size=cfg.getint(model_name, "vocab_size"),
+                start_id=cfg.getint(model_name, "start_id"),
+                end_id=cfg.getint(model_name, "end_id"),
+                weights_data_type=cfg.get(model_name, "weight_data_type"),
+                layernorm_eps=cfg.getfloat(model_name, 'layernorm_eps'),
+                rotary_embedding_dim=cfg.getint(model_name, 'rotary_embedding'),
+                inter_size=cfg.getint(model_name, 'inter_size'),
+                inference_data_type=inference_data_type)
+        else:
+            inference_data_type = self.dtype
+            if inference_data_type == None:
+                inference_data_type = LYRA_LLAMA_PARAM.weights_data_type
+            model_args = dict(head_num=LYRA_LLAMA_PARAM.num_heads,
+                              size_per_head=LYRA_LLAMA_PARAM.size_per_head,
+                              vocab_size=LYRA_LLAMA_PARAM.vocab_size,
+                              start_id=LYRA_LLAMA_PARAM.start_id or tokenizer.bos_token_id,
+                              end_id=LYRA_LLAMA_PARAM.end_id or tokenizer.eos_token_id,
+                              layer_num=LYRA_LLAMA_PARAM.num_layers,
+                              tensor_para_size=LYRA_LLAMA_PARAM.tensor_para_size,
+                              weights_data_type=LYRA_LLAMA_PARAM.weights_data_type,
+                              layernorm_eps=LYRA_LLAMA_PARAM.layernorm_eps,
+                              rotary_embedding_dim=LYRA_LLAMA_PARAM.rotary_embedding,
+                              inter_size=LYRA_LLAMA_PARAM.inter_size,
+                              inference_data_type=inference_data_type)
+
+        # update common parameters
+        model_args.update(dict(
+            lib_path=LIB_SO_PATH,
+            model_path=self.model_path,
+            max_seq_len=0,  # for position seq embedding
+            pipeline_para_size=LYRA_LLAMA_PARAM.pipeline_para_size,
+            use_gptj_residual=LYRA_LLAMA_PARAM.use_gptj_residual,
+            # shared_contexts_ratio=LYRA_LLAMA_PARAM.shared_contexts_ratio,
+        ))
+
+        print('[FT][INFO] Load Our FT Highly Optimized ChatGLM6B model')
+        for k, v in model_args.items():
+            print(f' - {k.ljust(25, ".")}: {v}')
+
+        # Check sanity and consistency between the model and tokenizer.
+        checklist = ['head_num', 'size_per_head', 'vocab_size', 'layer_num',
+                     'tensor_para_size', 'tensor_para_size', 'weights_data_type']
+        if None in [model_args[k] for k in checklist]:
+            none_params = [p for p in checklist if model_args[p] is None]
+            print(f'[FT][WARNING] Found None parameters {none_params}. They must '
+                  f'be provided either by config file or CLI arguments.')
+        if model_args['start_id'] != tokenizer.bos_token_id:
+            print('[FT][WARNING] Given start_id is not matched with the bos token '
+                  'id of the pretrained tokenizer.')
+        if model_args['end_id'] not in (tokenizer.pad_token_id, tokenizer.eos_token_id):
+            print('[FT][WARNING] Given end_id is not matched with neither pad '
+                  'token id nor eos token id of the pretrained tokenizer.')
+
+        print(f'Loading tokenizer from {self.model_path}')
+        model = LLaMAModel(**model_args)
+
+        return model, tokenizer
+
+    def generate(self, prompts: typing.List[str] | str,
+                 output_length: int = 512,
+                 beam_width: int = 1,
+                 top_k: typing.Optional[torch.IntTensor] = 1,
+                 top_p: typing.Optional[torch.FloatTensor] = 1.0,
+                 beam_search_diversity_rate: typing.Optional[torch.FloatTensor] = 0.0,
+                 temperature: typing.Optional[torch.FloatTensor] = 1.0,
+                 len_penalty: typing.Optional[torch.FloatTensor] = 0.0,
+                 repetition_penalty: typing.Optional[torch.FloatTensor] = 1.0,
+                 presence_penalty: typing.Optional[torch.FloatTensor] = None,
+                 min_length: typing.Optional[torch.IntTensor] = None,
+                 bad_words_list: typing.Optional[torch.IntTensor] = None,
+                 do_sample: bool = False,
+                 return_output_length: bool = False,
+                 return_cum_log_probs: int = 0):
+        #
+        if isinstance(prompts, str):
+            prompts = [prompts, ]
+
+        inputs = prompts
+
+        batch_size = len(inputs)
+        ones_int = torch.ones(size=[batch_size], dtype=torch.int32)
+        ones_float = torch.ones(size=[batch_size], dtype=torch.float32)
+
+        # we must encode the raw prompt text one by one in order to compute the length of the original text.
+        input_token_ids = [self.tokenizer(text, return_tensors="pt").input_ids.int().squeeze() for text in inputs]
+        input_lengths = torch.IntTensor([len(ids) for ids in input_token_ids])
+        # after got the length of each input text tokens. we can batchfy the input list to a tensor. padding the right.
+        input_token_ids = pad_sequence(input_token_ids, batch_first=True, padding_value=self.tokenizer.eos_token_id)
+
+        random_seed = None
+        if do_sample:
+            random_seed = torch.randint(0, 262144, (batch_size,), dtype=torch.long)
+
+        outputs = self.model(start_ids=input_token_ids,
+                             start_lengths=input_lengths,
+                             output_len=output_length,
+                             beam_width=beam_width,
+                             top_k=top_k * ones_int,
+                             top_p=top_p * ones_float,
+                             beam_search_diversity_rate=beam_search_diversity_rate * ones_float,
+                             temperature=temperature * ones_float,
+                             len_penalty=len_penalty * ones_float,
+                             repetition_penalty=repetition_penalty * ones_float,
+                             random_seed=random_seed,
+                             return_output_length=return_output_length,
+                             return_cum_log_probs=return_cum_log_probs)
+
+        if return_cum_log_probs > 0:
+            outputs = outputs[0]  # output_token_ids.
+
+        # Slice the generated token ids of the 1st beam result.
+        # output = input tokens + generated tokens.
+        output_token_ids = [out[0, length:].cpu()
+                            for out, length in zip(outputs, input_lengths)]
+
+        output_texts = self.tokenizer.batch_decode(
+            output_token_ids, skip_special_tokens=True)
+
+        return output_texts

lyra_llama/model.py

@@ -0,0 +1,156 @@
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import copy
+import os
+
+import numpy as np
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+
+class LLaMAModel(nn.Module):
+    def __init__(self,
+                 head_num, size_per_head, inter_size,
+                 vocab_size, rotary_embedding_dim,
+                 start_id, end_id, layer_num,
+                 max_seq_len, layernorm_eps,
+                 tensor_para_size, pipeline_para_size,
+                 use_gptj_residual,
+                 lib_path,
+                 model_path,
+                 inference_data_type: str = "fp16",
+                 weights_data_type: np.dtype = np.float16):
+        super().__init__()
+        self.head_num = head_num
+        self.size_per_head = size_per_head
+        self.inter_size = inter_size
+        self.vocab_size = vocab_size
+        self.rotary_embedding_dim = rotary_embedding_dim
+        self.start_id = start_id
+        self.end_id = end_id
+        self.max_seq_len = max_seq_len
+        self.layer_num = layer_num
+        self.use_gptj_residual = use_gptj_residual
+        self.layernorm_eps = layernorm_eps
+
+        self.tensor_para_size = tensor_para_size
+        self.pipeline_para_size = pipeline_para_size
+        self.build_model = False
+        self.weights_data_type = weights_data_type
+        self.inference_data_type = inference_data_type
+
+        assert torch.cuda.is_available(), "CUDA is required for this model."
+
+        assert head_num % tensor_para_size == 0, "head_num must be a multiple of tensor_para_size."
+        assert layer_num % pipeline_para_size == 0, "layer_num must be a multiple of pipeline_para_size."
+
+        # Load the C++ model into Pytorch model.
+        torch.classes.load_library(os.path.abspath(lib_path))
+
+        # Prepare for tensor/pipeline parallel
+        try:
+            dist.init_process_group(backend='mpi')
+        except:
+            print("[INFO] WARNING: Have initialized the process group")
+        self.rank = dist.get_rank()
+        self.device_count = torch.cuda.device_count()
+        self.device = self.rank % self.device_count
+        torch.cuda.set_device(self.device)
+
+        world_size = dist.get_world_size()
+        # print(tensor_para_size * pipeline_para_size)
+        assert world_size == tensor_para_size * pipeline_para_size, "tensor_para_size * pipeline_para_size must be equal to world_size."
+
+        self.tensor_para_rank = self.rank % self.tensor_para_size
+        self.pipeline_para_rank = self.rank // self.tensor_para_size
+
+        self.model = torch.classes.FasterTransformer.LlamaOp(
+            self.head_num,
+            self.size_per_head,
+            self.inter_size,
+            self.layer_num,
+            self.vocab_size,
+            self.rotary_embedding_dim,
+            self.layernorm_eps,
+            self.start_id,
+            self.end_id,
+            self.tensor_para_size,
+            self.pipeline_para_size,
+            self.max_seq_len,
+            self.use_gptj_residual,
+            model_path,
+            inference_data_type)
+
+        self.build_model = True
+
+    def forward(self,
+                start_ids: torch.Tensor,
+                start_lengths: torch.Tensor,
+                output_len,
+                beam_width=1,
+                top_k: torch.Tensor = None,
+                top_p: torch.Tensor = None,
+                beam_search_diversity_rate: torch.Tensor = None,
+                temperature: torch.Tensor = None,
+                len_penalty: torch.Tensor = None,
+                repetition_penalty: torch.Tensor = None,
+                random_seed: torch.Tensor = None,
+                return_output_length=False,
+                return_cum_log_probs=0):
+        if not self.build_model:
+            self.cuda()
+        input_len = start_ids.size(1)
+        assert input_len > 0, "input len must be larger than zero. For an unconditional case, use start_id as the first token."
+
+        # Inputs to device
+        input_ids = start_ids.cuda(self.device)
+        input_lengths = start_lengths.cuda(self.device)
+        # outputs: output_ids, output_lengths, output_cum_log_probs (optional)
+        outputs = self.model.forward(input_ids,
+                                     input_lengths,
+                                     output_len,
+                                     beam_width,  # optional, can be None
+                                     top_k,  # optional, can be None
+                                     top_p,  # optional, can be None
+                                     beam_search_diversity_rate,  # optional, can be None
+                                     temperature,  # optional, can be None
+                                     len_penalty,  # optional, can be None
+                                     repetition_penalty,  # optional, can be None
+                                     random_seed,  # optional, can be None
+                                     return_cum_log_probs)  # optional, can be None
+
+        if return_cum_log_probs == 0:
+            output_ids, output_lengths = outputs
+        else:
+            output_ids, output_lengths, output_cum_log_probs = outputs
+        if return_output_length:
+            if return_cum_log_probs > 0:
+                return output_ids, output_lengths, output_cum_log_probs
+            else:
+                return output_ids, output_lengths
+        else:
+            return output_ids
+
+    def set_input_tensor(self, input_tensor):
+        """Set input tensor to be used instead of forward()'s input.
+
+        When doing pipeline parallelism the input from the previous
+        stage comes from communication, not from the input, so the
+        model's forward_step_func won't have it. This function is thus
+        used by internal code to bypass the input provided by the
+        forward_step_func"""
+        self.input_tensor = input_tensor

models/config.ini

@@ -0,0 +1,13 @@
+[llama]
+model_name = ziya-llama
+head_num = 40
+size_per_head = 128
+inter_size = 13824
+num_layer = 40
+rotary_embedding = 128
+layernorm_eps = 1e-06
+vocab_size = 39424
+start_id = 1
+end_id = 2
+weight_data_type = fp16
+tensor_para_size = 1

models/special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<human>",
+    "<bot>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

models/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fd7e445833dd0889206aba242c2a51ecbae2437fd328d1759a35475fd8c0423
+size 588619

models/tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

requirements.txt

@@ -0,0 +1,4 @@
+transformers
+numpy
+setuptools
+torch