Running stateful inference on Kubernetes without tears

Most inference tutorials assume your model loads in milliseconds. Real models take 30–120 seconds to load weights onto the GPU. That single fact reshapes how you should schedule them.

Treat model load as a first-class lifecycle event

Use a readinessProbe that only passes once weights are resident and a warmup inference has run. Otherwise the Service will route traffic to a pod that returns 503s.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 20
  periodSeconds: 5
  failureThreshold: 30

Drain gracefully

• Set terminationGracePeriodSeconds high enough to finish in-flight requests.

• Trap SIGTERM, stop accepting new work, then exit.

• Pre-pull images with a DaemonSet so cold nodes don't add minutes.

Do these three things and rolling a new model version becomes a non-event.