MLMei Lin
Cutting LLM serving costs 60% with continuous batching
How vLLM's paged attention and dynamic batching transformed our inference economics.
CR
Carlos Ruiz
Author@carlos
GPU performance nerd
Squeezing every last token/sec out of accelerators. CUDA, MIG, and kernels.
2
Stories
364
Views
16
Likes
3
Followers
Contributions
JanFebMarAprMayJunJul
11 contributions in the last 6 monthsLessMore
DODaniel Okoro
Running stateful inference on Kubernetes without tears
StatefulSets, warm model caches, and graceful draining for GPU pods.
TBTom Becker
Edge inference: serving models at the CDN
Latency budgets when 50ms of network is unacceptable.
AKAisha Khan
The four golden signals for model inference
Latency, traffic, errors, saturation — adapted for GPU serving.
TBTom Becker
Building a model gateway with Envoy and gRPC
Routing, retries, and rate limits in front of your inference fleet.
DODaniel Okoro
Autoscaling inference with KEDA and custom metrics
Scale on queue depth, not CPU. Your wallet will thank you.