How to Make LLMs 3X Faster

Text generation in language models operates autoregressively, producing content one token at a time through sequential forward passes across all model layers. Because each new token strictly depends on the presence of all previous tokens, forward passes cannot be executed simultaneously without breaking coherence. Consequently, total generation duration equals the token count multiplied by the per-pass latency, which scales with model size. Modern inference systems mitigate the computational burden of subsequent passes by using a KV cache to store prior attention states, though the requirement for one pass per token remains.

During model inference, a forward pass spends the majority of its execution time moving data rather than computing arithmetic. In token generation, model weights must be transferred from VRAM to compute units for each single token, resulting in low compute utilization between 20 and 40 percent while the memory bus runs near capacity. In contrast, prompt processing achieves 90 to 95 percent utilization because weights are reused across thousands of input tokens. Consequently, increasing memory bandwidth accelerates token generation significantly more than adding raw compute capacity.

Transformers can evaluate an entire sequence simultaneously in a single forward pass, generating a next-token prediction for every position at once. Causal masking within the attention mechanism ensures each position only attends to preceding tokens, maintaining the validity of autoregressive conditioning. Consequently, candidate tokens can be verified in parallel during a single pass rather than sequentially, mirroring the computational efficiency of prompt processing.

The draft and verify framework accelerates language model inference by combining a small draft model with a large target model. In each round, the draft model serially generates a sequence of candidate tokens, which the target model verifies simultaneously in a single forward pass. Tokens are accepted sequentially until the first mismatch, at which point the target model's already-computed prediction is retained as an additional correct token. This setup bounds the worst-case throughput to that of standard single-token decoding while offering significant speedups with draft lengths typically set between 3 and 5.

Speculative decoding generates text with the exact same statistical properties as the target model operating independently, enforced via an acceptance rule. In greedy decoding, candidate tokens are accepted only if they match the target model's top prediction. When sampling with randomness, candidate acceptance depends on the probability distribution comparison between the target and draft models, with rejected tokens resampled from an adjusted distribution. The resulting token probabilities depend strictly on the target model, with deviations occurring only due to inherent sampling variance or numerical precision limits.

In candidate-token acceleration approaches, the speedup is primarily dictated by the acceptance rate, which represents the fraction of proposed tokens preserved by the target model. Repetitive or structured tasks like code generation and summarization yield high acceptance rates, whereas open-ended tasks like creative writing result in frequent model divergence and lower acceptance. Higher sampling temperatures lower acceptance further by flattening output probability distributions, with the process becoming inefficient when acceptance drops below roughly 50 percent. Demonstrating this in practice, DeepSeek documented an 80 to 90 percent acceptance rate for the second predicted token during production serving of DeepSeek-V3, which yielded roughly 1.8x generation throughput.

Speculative drafting relies on four primary mechanisms to obtain fast, low-cost token predictions: deploying a separate smaller sibling model, adding multi-token prediction heads to the target architecture, executing a lower-precision or pruned variant of the same weights, or retrieving matching sequences from prior context. Each option introduces distinct trade-offs between hardware overhead, training requirements, implementation complexity, and contextual applicability. Tokenizer alignment strictly dictates draft compatibility, often presenting a higher barrier than model quality differences. Ultimately, all four drafting strategies necessitate sufficient spare compute capacity during serving to provide a net speedup.

The speedup provided by speculative decoding depends significantly on server load and operating concurrency. As concurrent requests increase, compute units approach saturation and verification must compete with incoming requests, causing latency gains to diminish and potentially fall below baseline throughput. Serving systems such as vLLM manage this tradeoff by dynamically reducing draft length or disabling speculation beyond certain batch sizes. Additionally, because the technique targets token generation rather than prompt evaluation, it offers minimal latency benefits for workloads characterized by long prompts and short outputs.

Speculative decoding reorganizes the timing of token generation rather than reducing the total work executed by the target model. It leverages the transformer architecture's ability to evaluate multiple candidate tokens in parallel for roughly the cost of a single token. When drafted tokens are rejected, the verification pass still produces a correct token at the point of mismatch, preventing wasted computation. Consequently, baseline output quality is maintained by design through the acceptance rule without additional tuning.