Engineering stories, reviewed by editors

The craft of serving models at scale

Deep, practical writing on Kubernetes, GPUs, autoscaling, and the economics of machine-learning inference — from engineers who run it in production.

Featured

How vLLM's paged attention and dynamic batching transformed our inference economics.

MLMei LinJul 51m46

StatefulSets, warm model caches, and graceful draining for GPU pods.

Routing, retries, and rate limits in front of your inference fleet.

TBTom BeckerJun 161m289

When to slice an A100 into seven, and when not to.

Jun 261 min22510Kubernetes

Latency, traffic, errors, saturation — adapted for GPU serving.

Jun 211 min2487ML Inference

Scale on queue depth, not CPU. Your wallet will thank you.

Jun 101 min909Kubernetes

Draft models, acceptance rates, and the latency wins that survived contact with reality.

Jun 31 min1366ML Inference

Error budgets when your dependency is a probabilistic model.

May 271 min584Security

INT8, FP8, and AWQ — measuring what you actually lose.

May 201 min1396ML Inference

Latency budgets when 50ms of network is unacceptable.

May 131 min886ML Inference