Topic

ML Inference

7 stories

How vLLM's paged attention and dynamic batching transformed our inference economics.

Jul 51 min4610ML Inference

StatefulSets, warm model caches, and graceful draining for GPU pods.

Jul 11 min1278Kubernetes

Latency, traffic, errors, saturation — adapted for GPU serving.

Jun 211 min2487ML Inference

Routing, retries, and rate limits in front of your inference fleet.

Jun 161 min2897ML Inference

Draft models, acceptance rates, and the latency wins that survived contact with reality.

Jun 31 min1366ML Inference

INT8, FP8, and AWQ — measuring what you actually lose.

May 201 min1396ML Inference

Latency budgets when 50ms of network is unacceptable.

May 131 min886ML Inference