File size: 193 Bytes
21a28bf |
1 2 3 4 5 6 7 8 9 10 |
---
tags:
- fp8
- vllm
---
Run with `vllm==0.6.2` on 4xH100:
```
vllm serve neuralmagic/Llama-3.2-90B-Vision-Instruct-FP8-dynamic --enforce-eager --max-num-seqs 16 --tensor-parallel-size 4
``` |