Autoscaling inference with KEDA and custom metrics

CPU-based autoscaling is meaningless for GPU inference. The right signal is how many requests are waiting.

triggers:
  - type: prometheus
    metadata:
      query: sum(inference_queue_depth)
      threshold: "10"

Scale to zero, carefully

Scaling to zero saves money overnight, but cold starts hurt. Keep one warm replica for latency-sensitive tiers and let batch tiers go cold.

• Warm pool for interactive traffic.

• Cold scale-to-zero for batch.

• Pre-pull images to cut cold-start minutes.