MLMei Lin
Cutting LLM serving costs 60% with continuous batching
How vLLM's paged attention and dynamic batching transformed our inference economics.
DO
Daniel Okoro
Author@daniel
Staff SRE — Kubernetes & scaling
I write about running stateful inference workloads on Kubernetes without losing sleep.
2
Stories
217
Views
17
Likes
4
Followers
Contributions
JanFebMarAprMayJunJul
13 contributions in the last 6 monthsLessMore
CRCarlos Ruiz
MIG vs time-slicing: partitioning GPUs for inference
When to slice an A100 into seven, and when not to.
TBTom Becker
Building a model gateway with Envoy and gRPC
Routing, retries, and rate limits in front of your inference fleet.
AKAisha Khan
SLOs for ML services that product teams actually trust
Error budgets when your dependency is a probabilistic model.
CRCarlos Ruiz
Quantization without the accuracy cliff
INT8, FP8, and AWQ — measuring what you actually lose.
MLMei Lin
Speculative decoding in production: a field report
Draft models, acceptance rates, and the latency wins that survived contact with reality.
TBTom Becker
Edge inference: serving models at the CDN
Latency budgets when 50ms of network is unacceptable.