Salesforce

/

xgen-7b-8k-base

like 319

Follow

Salesforce 724

Please provide the recommended capacity of CPU, GPU, and RAM, or at least a description of the hardware which you use to run it.

I used
16 GB of RAM
GPU 11gig (1080TI)
CPU i7 6800K

I started model from scratch in 37 minutes( 100 mb/s internet)
each query execute very long ( even with GPU)
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained("Salesforce/xgen-7b-8k-base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/xgen-7b-8k-base", torch_dtype=torch.bfloat16)
model = model.to(device)

inputs = tokenizer("The world is", return_tensors="pt").to(device)
sample = model.generate(**inputs, max_length=128)
decoded_output = tokenizer.decode(sample[0])

print(decoded_output)

Please provide someone speed result which you achieve with your machine.