README.md

metadata

license: creativeml-openrail-m
language:
  - en
tags:
  - LLM
  - tensorRT
  - chatGLM

Model Card for lyraChatGLM

lyraChatGLM is currently the fastest chatGLM-6B available, as far as we know, it is also the fisrt accelerated version of chatGLM-6B.

The inference speed of lyraChatGLM is 10x faster than the original version, and we're still working to improve the performance.

Among its main features are:

• weights: original ChatGLM-6B weights released by THUDM.
• device: lyraChatGLM is mainly based on FasterTransformer compiled for SM=80 (A100, for example).

Speed

test environment

• device: Nvidia A100 40G
• img size: 512x512
• percision:fp16
• steps: 30
• solver: LMSD

text2img

Model Sources

• Repository: [https://huggingface.co/THUDM/chatglm-6b]

Uses

from faster_chat_glm import GLM6B, FasterChatGLM

# kernel for chat model.
kernel = GLM6B(plan_path=plan_path,
               batch_size=BATCH_SIZE,
               num_beams=1,
               use_cache=USE_CACHE,
               num_heads=32,
               emb_size_per_heads=128,
               decoder_layers=28,
               vocab_size=150528,
               max_seq_len=MAX_OUT_LEN)

chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()

# generate
sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)

Demo output

text2img

img2img

Environment

• hardware: Nvidia Ampere architecture (A100) or compatable
• docker image avaible: https://hub.docker.com/r/bigmoyan/lyra_aigc/tags

docker pull bigmoyan/lyra_aigc:v0.1

Citation

@Misc{lyraChatGLM2023,
  author =       {Kangjian Wu, Zhengtao Wang, Bin Wu},
  title =        {lyaraChatGLM: Accelerating chatGLM by 10x+},
  howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
  year =         {2023}
}

Report bug

• start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
• report bug with a [bug] mark in the title.