DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
vLLM Constrained Decoding: Fix Tool Call Schema Errors
rammehta1899 profile

vLLM Constrained Decoding: Fix Tool Call Schema Errors

#vllm #llmserving #agenticworkflows #constraineddecoding
Comments
1 min read
vLLM prefix caching on vs off: 7.5x the throughput with a 4,000-token system prompt on one L40S
runcrate profile

vLLM prefix caching on vs off: 7.5x the throughput with a 4,000-token system prompt on one L40S

#vllm #llm #gpu #performance
Comments
6 min read
vLLM FP8 vs bf16 on an H100: 1.4x the throughput, and what it did to GSM8K accuracy
runcrate profile

vLLM FP8 vs bf16 on an H100: 1.4x the throughput, and what it did to GSM8K accuracy

#vllm #llm #gpu #benchmark
Comments
5 min read
Your Local LLM Isn't Bad at Tool Calling, Your Pipeline Is: Fixing Tool-Call Accuracy in vLLM
eme_gug_0821b41b948be6516 profile

Your Local LLM Isn't Bad at Tool Calling, Your Pipeline Is: Fixing Tool-Call Accuracy in vLLM

#llm #vllm #python #ai
Comments
7 min read
vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100
runcrate profile

vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100

#vllm #llm #gpu #benchmark
1
Comments
5 min read
Two vLLM servers on one GPU: the startup check that bites, and the fix
qaisermehdi profile

Two vLLM servers on one GPU: the startup check that bites, and the fix

#vllm #llm #gpu #mlops
Comments 1
2 min read
llm-d vs Plain vLLM: A Practical Kubernetes Inference Quickstart
jamilxt profile

llm-d vs Plain vLLM: A Practical Kubernetes Inference Quickstart

#llmd #vllm #kubernetes #ai
1
Comments
8 min read
KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
wolfsea2357 profile

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

#ai #machinelearning #vllm #llm
Comments
14 min read
Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
wolfsea2357 profile

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

#ai #machinelearning #vllm #performance
Comments
12 min read
Why LLM Inference Engines Are Fast: Continuous Batching, PagedAttention and Prefix Caching Explained with vLLM
eme_gug_0821b41b948be6516 profile

Why LLM Inference Engines Are Fast: Continuous Batching, PagedAttention and Prefix Caching Explained with vLLM

#llm #vllm #inference #performance
Comments
7 min read
A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128
qaisermehdi profile

A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128

#llm #gpu #vllm #benchmark
Comments
2 min read
Why reasoning_effort behaves differently across providers
orca_forge profile

Why reasoning_effort behaves differently across providers

#llm #openai #vllm #api
Comments 1
5 min read
vLLM's --api-key leaves /invocations, /tokenize and /pause open. Lock it down with an allowlist proxy
munimv2 profile

vLLM's --api-key leaves /invocations, /tokenize and /pause open. Lock it down with an allowlist proxy

#vllm #llm #security #abotwrotethis
Comments
3 min read
Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost
mtahir27 profile

Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost

#modeldistillation #quantization #vllm #awq
Comments
10 min read
Day 16: Local Model Execution — Running Open-Weight LLMs Securely with Ollama and vLLM
suresh_kumar_de3920bedd1c profile

Day 16: Local Model Execution — Running Open-Weight LLMs Securely with Ollama and vLLM

#ai #ollama #vllm #30dayaiseries
Comments
4 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.