Topic

Kubernetes

3 stories

Running stateful inference on Kubernetes without tears

StatefulSets, warm model caches, and graceful draining for GPU pods.

Jul 11 min1278Kubernetes

MIG vs time-slicing: partitioning GPUs for inference

When to slice an A100 into seven, and when not to.

Jun 261 min22510Kubernetes

Autoscaling inference with KEDA and custom metrics

Scale on queue depth, not CPU. Your wallet will thank you.

Jun 101 min909Kubernetes