LLM Inference Frameworks - Category - Naifan Li's Bloghttps://blog.omagiclee.com/categories/llm-inference-frameworks/LLM Inference Frameworks - Category - Naifan Li's BlogHugo -- gohugo.ioen-usThu, 27 Nov 2025 21:07:50 +0800vLLM: Easy, fast, and cheap LLM inference and servinghttps://blog.omagiclee.com/posts/toolkits/llm-inference-engines/vllm/Thu, 27 Nov 2025 21:07:50 +0800Naifan Lihttps://blog.omagiclee.com/posts/toolkits/llm-inference-engines/vllm/Docs ยท GitHub

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.

  • Run open-source models on vLLM
  • Build appplications with vLLM
  • Build vLLM

vLLM is fast with:

  • State-of-the-art serving throughput
  • Efficient management of attention key and value memory with PagedAttention
  • Continuous batching of incoming requests
  • Fast model execution with CUDA/HIP graph
  • Quantization: GPTQ, AWQ, INT4, INT8, and FP8
  • Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
  • Speculative decoding
  • Chunked prefill

vLLM is flexible and easy to use with:

]]>