Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Paper
•
2407.00079
•
Published
Research projects on top of vLLM
Papers cited in https://blog.vllm.ai/2024/07/25/lfai-perf.html
-
-
Llumnix: Dynamic Scheduling for Large Language Model Serving
Paper • 2406.03243 • Published -
CacheGen: Fast Context Loading for Language Model Applications
Paper • 2310.07240 • Published • 1 -
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Paper • 2405.04437 • Published • 3 -
Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services
Paper • 2404.16283 • Published -
Efficiently Programming Large Language Models using SGLang
Paper • 2312.07104 • Published • 7