CRCarlos Ruiz
Quantization without the accuracy cliff
INT8, FP8, and AWQ — measuring what you actually lose.
ML
Mei Lin
Author@meilin
ML Infrastructure Engineer
Serving large models efficiently. KServe, vLLM, and the dark art of batching.
2
Stories
182
Views
16
Likes
6
Followers
Contributions
JanFebMarAprMayJunJul
11 contributions in the last 6 monthsLessMore
DODaniel Okoro
Running stateful inference on Kubernetes without tears
StatefulSets, warm model caches, and graceful draining for GPU pods.
TBTom Becker
Edge inference: serving models at the CDN
Latency budgets when 50ms of network is unacceptable.
AKAisha Khan
The four golden signals for model inference
Latency, traffic, errors, saturation — adapted for GPU serving.
CRCarlos Ruiz
MIG vs time-slicing: partitioning GPUs for inference
When to slice an A100 into seven, and when not to.
DODaniel Okoro
Autoscaling inference with KEDA and custom metrics
Scale on queue depth, not CPU. Your wallet will thank you.