Speculative decoding in production: a field report
Draft models, acceptance rates, and the latency wins that survived contact with reality.
ML
Mei Lin
Jun 3, 2026·1 min read·136
Speculative decoding promises faster generation by letting a small draft model propose tokens a big model verifies in parallel. Here's what actually happened when we shipped it.
Acceptance rate is everything
If the draft model's tokens get rejected, you pay for both models and gain nothing. We saw 2.1x speedups on code, 1.3x on creative writing.
engine = LLM(model="big", speculative_model="small",
num_speculative_tokens=5)Tune the number of speculative tokens per workload — there is no universal best value.
ML
Written by
Mei LinML Infrastructure Engineer
Serving large models efficiently. KServe, vLLM, and the dark art of batching.
6 followers · 2 stories