The “Why read it” summaries are generated by AI. However, the readings themselves are selected and curated by the instructor.
Speculative decoding spends inexpensive work to avoid expensive serial model steps while preserving the target model's output distribution. The readings answer five questions in order. First, why is verifying several guessed tokens in one pass exact? Second, where do the guesses come from? Three groups answer it: drafters built from a separate small model or from the target's own layers, heads, and features; drafters that propose a whole block in one forward pass; and drafters that copy text the system has already seen. Third, how does a verifier check a tree of candidates, and when may it accept tokens the target would not have produced? Fourth, when does speculation pay inside a serving engine as batch size, context length, and expert routing change? Fifth, how are drafters trained for each new target model? Start with the two independent formulations of the method, then treat every later proposal as a change to draft cost, acceptance rate, verification shape, or scheduling overhead.
| Paper | Year | Why read it |
|---|---|---|
| Guess ahead, verify in parallel | ||
| speculative decodingFast Inference from Transformers via Speculative Decoding | 2022 |
Background. An autoregressive model emits one token per forward pass, and at small batch sizes that pass is bound by memory bandwidth: the weights are read from HBM to produce a single token. Problem. The sequential dependency, not the arithmetic, sets the latency. You cannot start token n+1 before token n exists, so a response costs a fixed number of passes however much of the accelerator each pass leaves idle. Verifying several tokens at once would be nearly free, but you do not have them yet. Key idea. Let a cheap model guess the next several tokens, run the target model once over the whole guessed block, and accept a prefix using a rejection-sampling rule. The rule is constructed so the accepted tokens carry exactly the target model's distribution, so only the failed guesses cost anything. Findings. On T5-XXL, speculative decoding runs 2-3× faster than the standard T5X implementation while producing identical outputs. Why it matters. It separates how aggressively you guess from whether the output changes, and the exactness argument is what makes speculation acceptable in production at all. Every later method in this group varies the drafting and reuses this verification step unchanged. Adoption. vLLM, SGLang, TensorRT-LLM, llama.cpp, and Hugging Face Transformers all ship speculative-decoding or assisted-generation implementations of this verify step. |
| speculative samplingAccelerating Large Language Model Decoding with Speculative Sampling | 2023 |
Background. The entry above states the propose-and-verify scheme and the rejection-sampling argument for why the output distribution survives it. This is the concurrent DeepMind treatment of the same scheme. Problem. Whether the accepted tokens really carry the target model's distribution rests on a derivation, not on a measured speedup. A correctness argument that only one group has written down is hard to trust and harder to build on. Key idea. Derive the same accept-or-resample rule independently, setting the argument up differently, and report it on large models at serving scale. Reading the two derivations together shows which parts of the rule are forced by preserving the distribution and which are free choices. Findings. On the 70B-parameter Chinchilla model in a distributed setup, speculative sampling accelerates decoding by 2-2.5× without changing sample quality. Why it matters. Two independent derivations landing on one acceptance rule is the signal that the rule is the constraint rather than an implementation detail. Everything later in this group changes where the guesses come from and leaves the rule alone. |
| blockwise parallel decodingfor Deep Autoregressive Models | 2018 |
Background. In 2018 autoregressive Transformers were used for machine translation, and they already had the constraint decoding has now: the parallelism available while training disappears at inference time. Problem. Greedy decoding of an output of length n takes n sequential passes even though each pass leaves the accelerator mostly idle. The non-autoregressive proposals of the period bought parallelism by changing the output, which translation quality would not absorb. Key idea. Add auxiliary output heads that predict the next several tokens from the same hidden state, verify the whole block in one pass of the base model, and keep the longest prefix that matches what greedy decoding would have produced. Propose in parallel, verify in one pass, accept only what agrees. Findings. The evaluated translation and image-super-resolution models cut decoding iterations by up to 2× without quality loss. Allowing a small quality decrease cuts iterations by up to 7×, while the fastest wall-clock configuration runs up to 4× faster than greedy decoding. Why it matters. It is the ancestor of this entire group, four years before speculative decoding, and it shows the propose-and-verify shape needs no second model. The later split between separate draft models and extra heads on the target model is already present here. |
| Where the guesses come from | ||
| DistillSpecImproving Speculative Decoding via Knowledge Distillation | 2023 |
Background. The first speculative decoding papers draft with a smaller model from the target's family, such as T5-small for T5-XXL. That model is trained to be a good language model in its own right, not to agree with one particular target. Problem. The speedup depends on the acceptance rate, which measures agreement with the target rather than the drafter's own quality. A small model that predicts text well can still disagree with the target often enough to cap the speedup, and making the drafter larger to improve it also makes every draft token more expensive. Key idea. Distill the drafter from the target. The token-level acceptance rate equals one minus the total variation distance between the target and draft distributions, so training the drafter toward the target raises acceptance directly. Two choices matter: train on sequences the drafter samples itself (on-policy data), scored by the target, and pick the divergence (forward KL, reverse KL, Jensen-Shannon, or total variation) to suit the task and the decoding strategy. Findings. Across the evaluated T5 and decoder-only model pairs, DistillSpec runs 10-45% faster than speculative decoding with the undistilled drafter, under both greedy decoding and sampling. When a smaller target is first distilled from the large model and DistillSpec then trains its drafter, decoding latency falls 6-10× relative to standard decoding without distillation, with a small performance drop; this setting no longer reproduces the large model's outputs exactly. Why it matters. It turns acceptance rate into a training objective rather than a property of whichever small model is available. The divergence results add a practical rule: the objective that maximizes acceptance depends on whether you serve with greedy decoding or with sampling, so a drafter trained for one is not automatically the best for the other. |
| self-speculativeDraft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding | 2023 |
Background. Speculative decoding is lossless because the full model verifies every token; the drafter itself can be anything cheap. Until this paper the cheap thing was a separate small model, which you have to obtain, align, and hold in memory. Problem. A suitable small sibling often does not exist, and when it does it is extra weights resident beside the target for the life of the deployment. What you want is a cheap approximation of the model you already loaded, with nothing new trained and nothing new stored. Key idea. Draft by running the target model with a subset of its intermediate layers skipped. The subset is selected once, offline, and the surviving layers act as a fast approximate model that proposes tokens; the full model then verifies them under the standard acceptance rule, so the output distribution is unchanged. Findings. Across Llama-2 and its variants, self-speculative decoding reports speedups of up to 1.99× with identical outputs, no extra training, and no additional model memory. Why it matters. It isolates the assumption underneath all of speculation, that a degraded version of the model agrees with it most of the time, and gets that degraded version free from the same weights. It also ties speculative decoding to early exit and depth pruning, usually treated as separate work. |
| LayerSkipEnabling Early Exit Inference and Self-Speculative Decoding | 2024 |
Background. Draft & Verify, the self-speculative entry above, drafts by skipping layers of an unmodified model. Early exit is the other way to draft from the target's own layers: run only the first E layers and decode their output through the language-model head. Problem. In a normally trained model the intermediate layers are not trained to feed the language-model head, so tokens predicted after an early exit are often wrong and few drafts survive verification. A separate drafter avoids this problem but brings its own weights and its own KV cache. Key idea. Train the model so that its early layers can predict. Layer dropout rises with depth, so the model learns to rely less on its later layers, and an early-exit loss trains every layer through one language-model head shared across layers. At inference, the first E layers draft tokens autoregressively and the remaining layers verify them in one pass. Drafting and verification share one KV cache, and an exit query cache keeps the query at the exit layer, so verification resumes there instead of recomputing the first E layers. Findings. Relative to autoregressive decoding, self-speculative decoding runs up to 2.16× faster on CNN/DM summarization (Llama2 7B pretrained with the recipe), 1.82× on HumanEval coding (Llama1 7B fine-tuned on code), and 2.0× on TOPv2 semantic parsing (a 1.5B Llama model fine-tuned on that task). Why it matters. One set of weights and one KV cache serve both drafting and verification, so speculation adds no second model to load or cache. The cost moves to training. The model must be pretrained or fine-tuned with the recipe, and the recipe can lower the full model's accuracy: continual pretraining of Llama3 8B with it lowered last-layer MMLU from 66.5 to 60.5 and GSM8K from 54.2 to 45.0. Adoption. Hugging Face Transformers runs early-exit self-speculative decoding on LayerSkip checkpoints. |
| SWIFTOn-the-Fly Self-Speculative Decoding for LLM Inference Acceleration | 2024 |
Background. Draft & Verify chooses the layers to skip offline, by Bayesian optimization for each model and task, and LayerSkip retrains the model so its early layers can draft. Both fix the drafter before serving begins. Problem. The offline search is slow, and its result does not transfer between tasks. The paper reports that 1,000 Bayesian-optimization iterations take nearly 7.5 hours for LLaMA-2-13B and 20 hours for LLaMA-2-70B on an A6000 server, and that a layer set tuned for story generation falls from a 1.47× speedup to 1.01× on a reasoning task. Key idea. Choose the skipped layers during inference, from the current context. After a warm-up of 32 generated tokens, SWIFT proposes candidate layer sets, mostly by random search with a Bayesian-optimization proposal every 25 steps, and scores each by matchness: the fraction of recently generated tokens that the layer-skipped model reproduces when run over them in parallel. When the search ends, the best set drafts. Drafting stops early when the drafter's top-1 probability falls below a threshold, and uncertain positions add top-k alternatives that the target verifies in the same pass. Findings. On LLaMA-2 and CodeLLaMA models across summarization, math reasoning, story generation, and code generation, SWIFT runs 1.3-1.6× faster than autoregressive decoding while preserving the output distribution. Why it matters. It removes the last offline step from layer-skipping speculation: nothing is trained, nothing extra is stored, and the drafter adapts to each input stream. The search moves into serving instead, where it spends decoding steps on candidate layer sets before the drafter settles. |
| MedusaSimple LLM Inference Acceleration Framework with Multiple Decoding Heads | 2024 |
Background. Classic speculative decoding puts a second, smaller model in front of the target. That means training a draft model, keeping its tokenizer and its distribution aligned with the target, and finding memory and scheduling room for it in the serving stack. Problem. The second model is an operational cost that contributes nothing to the speedup itself. It has to be chosen per target model, loaded beside it, and batched with it, and for many targets no suitable small sibling exists at all. Key idea. Attach extra decoding heads to the target model, each trained to predict a token one position further ahead than the last. Their combinations form a small candidate tree that the target verifies in a single pass with tree attention, so the draft comes out of the model you already run and no second model is served. Findings. Medusa-1 accelerates the evaluated models by more than 2.2× without reducing generation quality. Jointly tuning the heads and backbone in Medusa-2 raises the reported speedup to 2.3-3.6×. Why it matters. It shows the drafter need not be a model. Capacity to predict future positions can live in a few heads on top of the existing network, which reduces the deployment cost of speculation from a second model to one extra set of small weights. Adoption. TensorRT-LLM and Hugging Face TGI ship Medusa heads, and vLLM carries a Medusa proposer. |
| EAGLESpeculative Sampling Requires Rethinking Feature Uncertainty | 2024 |
Background. Speculative decoding needs a cheap proposer whose guesses the target model usually accepts. The standard choice is a smaller independent model, which has to be trained separately, served alongside the target, and kept aligned with it. Problem. Drafting one token at a time from tokens alone discards the target model's own hidden state, and a token sequence does not fix what comes next: sampling introduces uncertainty that a token-level drafter cannot see. Acceptance stays low for the cost paid. Key idea. Draft in feature space rather than token space. One small autoregressive head sits on top of the target model and predicts its hidden features a step ahead, conditioned on the tokens already sampled so that sampling uncertainty is an input rather than a source of error. The model's own output layer turns those features into draft tokens. Findings. On Llama-2-Chat-70B, EAGLE reports a 2.7-3.5× latency speedup and 2× throughput while preserving the generated-text distribution. Why it matters. It reframes drafting as a question of what the target model already computes rather than what a second model can guess. The drafter is a head, not a model, so there is one set of weights to serve and the accuracy comes from reusing the target's representation. Adoption. vLLM, SGLang, and TensorRT-LLM ship EAGLE drafting, and Meta serves Llama 3 and Llama 4 models with EAGLE drafters in production. |
| EAGLE-2Faster Inference of Language Models with Dynamic Draft Trees | 2024 |
Background. Tree drafting proposes several candidate continuations at once and verifies the whole tree in one target-model pass. EAGLE and the tree methods before it use a tree whose shape is fixed in advance and identical at every decoding step. Problem. A fixed tree spends its budget uniformly, but acceptance is not uniform. How far a draft survives depends on the context, so some branches are worth extending several tokens and others are dead at the first position. A static shape spends verification on branches that will be rejected. Key idea. Build the tree at run time. The draft head's own confidence scores approximate each candidate's acceptance probability, so the tree expands where confidence is high and prunes where it is low, giving a different shape at every step with no change to the target model or the acceptance rule. Findings. Across three model series and six tasks, EAGLE-2 reports 3.05-4.26× speedups, which are 20-40% higher than EAGLE, while preserving the generated-text distribution. Why it matters. It separates two things speculative decoding usually conflates, how good the drafter is and how the draft budget is spent. The second turns out to be worth a large share of the speedup on its own, and it costs no extra training. |
| EAGLE-3Scaling up Inference Acceleration of Large Language Models via Training-Time Test | 2025 |
Background. EAGLE trains its draft head to predict the target model's features, and that recipe became the default drafter in the major serving engines, with correspondingly more data and compute available to train heads. Problem. The feature-prediction objective caps what more data can buy. A head constrained to reproduce one representation stops improving once it fits it, so additional training data no longer lengthens accepted drafts; the drafter saturates while the training budget keeps growing. Key idea. Drop the feature-prediction constraint. The head predicts tokens directly and consumes fused features from several layers of the target model rather than one, and it is trained under the multi-step conditions it meets at inference, the training-time test the title names. Draft quality then keeps improving as training data grows. Findings. EAGLE-3 reaches a speedup of up to 6.5× and improves over EAGLE-2 by about 1.4×. In SGLang at batch size 64, it raises throughput by 1.38× over the evaluated baseline. Why it matters. It makes the drafter something you scale rather than something you tune. Acceptance length becomes a function of training compute, which is a different engineering posture from choosing tree shapes and draft lengths by hand. Adoption. vLLM, SGLang, and TensorRT-LLM ship EAGLE-3 drafting. |
| EAGLE at MetaEfficient Speculative Decoding for Llama at Scale: Challenges and Solutions | 2025 |
Background. The EAGLE papers report speedups for a drafter running in research code. A production deployment runs the same algorithm inside a serving stack with tensor parallelism, paged KV caches, CUDA graphs, and large batches. Problem. Tree attention and multi-round speculative sampling are hard to run efficiently on GPUs at production scale. Large batches add a further problem: decoding becomes compute-bound, so drafting and verification no longer use free compute. The paper cites a vLLM measurement in which EAGLE's speedup falls from 1.3× at batch size 2 to 0.7× at batch size 48. Key idea. Engineer the whole path for Llama 3 and Llama 4. Training uses online distillation against the target's hidden states and logits and a three-layer dense drafter. Inference splits tree attention into a prefix pass and a suffix pass, compiles multi-round speculative sampling and parallelizes it across tensor-parallel ranks, chooses a precomputed static tree for each batch size, keeps paged KV caches for both models, captures CUDA graphs over the whole step, and quantizes the drafter's feed-forward layers to INT4. Findings. Llama 4 Maverick decodes at about 4 ms per token at batch size one on 8 H100 GPUs, 10% faster than the previously best known method. At large batch sizes, EAGLE speeds up production decoding by 1.4-2.0× over decoding without speculation. Why it matters. Whether speculation pays at large batch depends on the model and the context length, not on batch size alone. In the paper's measurements Llama 3.1 8B shows a larger speedup at large batch sizes, while the speedup for Llama 4 Maverick, at about 400B parameters, decreases as batch size grows, and at long context lengths attention dominates the step even at large batch. Most of the paper is systems engineering around a known drafter: synchronization, kernels, tree shapes, and cache layout. |
| DeepSeek-V3 MTPDeepSeek-V3 Technical Reportalso Oct 19 | 2024 |
Background. Medusa adds draft heads to a finished model and trains them afterwards, so the heads learn from a fine-tuning dataset rather than from the data the model was pretrained on. DeepSeek-V3 is a 671B-parameter mixture-of-experts model whose report describes its architecture, pretraining, and post-training together. Problem. Next-token prediction gives the model one training target per position. Predicting further ahead could give it a denser signal, but independent heads for future positions, as in Medusa, do not see the tokens that would precede the one they predict. Key idea. Add multi-token prediction (MTP) modules during pretraining. Each module predicts one additional future token from the previous depth's representation and the embedding of the next true token, so the causal chain is kept, and it shares the embedding and output head with the main model. The modules exist to improve training. At inference you can discard them, or keep them as a built-in drafter for speculative decoding. Findings. DeepSeek-V3 predicts the next two tokens through MTP. The second token is accepted 85-90% of the time across generation topics, and speculative decoding with it delivers 1.8× the tokens per second. Why it matters. The drafter is trained by the model's developer, on the pretraining corpus, at pretraining scale, and it ships inside the checkpoint. Speculative decoding then needs no separate draft-training effort for that model. Adoption. vLLM, SGLang, and TensorRT-LLM run DeepSeek-V3's MTP module as a speculative drafter. |
| Drafting a whole block in one pass | ||
| DFlashBlock Diffusion for Flash Speculative Decoding | 2026 |
Background. EAGLE-style drafters are autoregressive: drafting k tokens takes k sequential drafter passes. Diffusion language models generate many tokens in parallel, but as standalone models they trail autoregressive models in quality. Problem. Sequential drafting puts a floor under the drafting latency, so a longer draft costs proportionally more time. That cost limits how long a draft is worth proposing, and therefore limits the speedup. Key idea. Use a lightweight block diffusion model as the drafter. It generates the whole draft block in a single forward pass, conditioned on context features extracted from the target model. Verification is the standard lossless rule, so the diffusion model's quality gap affects only acceptance, not output. Findings. DFlash reports over 6× lossless acceleration across the evaluated models and tasks, and up to 2.5× higher speedup than EAGLE-3. Why it matters. It decouples draft length from drafting latency. The later entries in this group keep the one-pass draft and work on what it gives up, chiefly that positions in the block are predicted independently of each other. Adoption. vLLM, SGLang, and TensorRT-LLM ship DFlash drafting. |
| DominoDecoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding | 2026 |
Background. An autoregressive drafter models dependencies between draft tokens but pays sequential overhead. A parallel drafter such as DFlash removes that overhead but predicts each position without seeing the drafted tokens before it. Problem. Verification stops at the first rejected token, so a block whose positions do not agree with each other loses everything after the first disagreement. Parallel drafting trades that intra-block consistency for speed. Key idea. Split the two jobs. A parallel backbone produces preliminary distributions for the whole block, and a lightweight Domino head then refines them with prefix-dependent causal information. A base-anchored training curriculum first strengthens the backbone and then shifts the objective to the corrected distribution. Findings. On Qwen3 models, Domino reaches up to 5.49× end-to-end speedup under the Transformers backend and up to 5.8× throughput speedup under SGLang serving. Why it matters. It shows that causal modeling and sequential execution can be separated. A small amount of sequential computation after a parallel pass recovers much of the dependency that full autoregressive drafting provides. Adoption. SGLang ships a Domino drafter. |
| DSparkConfidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation | 2026 |
Background. Parallel drafters propose a long block of tokens in one forward pass, which is what makes drafting cheap enough to matter at scale. AdaServe, in the serving group below, makes the verification budget a per-request scheduling variable. Problem. Parallel drafts omit dependencies within the block, so acceptance decays along its suffix. Verifying the full block then spends scarce batch capacity on likely rejections. Key idea. Add a lightweight sequential module so the draft captures intra-block dependencies. Then choose each request's verification length from its estimated prefix survival probability and the engine's measured throughput under current load. Findings. Accepted length improves over autoregressive and parallel drafters. Under live DeepSeek-V4 traffic, per-user generation speed rises 60-85% over the MTP-1 baseline at matched throughput. Why it matters. Draft length reflects model quality; verification length reflects current serving capacity. Treating them separately turns speculation into a per-request scheduling decision. Adoption. vLLM, SGLang, and TensorRT-LLM ship DSpark drafting. |
| DFlash 2Keep Drafting Parallel | 2026 |
Background. DFlash predicts every position of the block independently. Domino and DSpark make the block consistent by adding a sequential module that rewrites each position's full-vocabulary distribution. Problem. The correct token is usually among the drafter's candidates even when its top pick is wrong. For a five-layer Qwen3-4B DFlash drafter on GSM8K, the top pick at the first position is right 85.4% of the time, but the right token is in the top 16 candidates 99.5% of the time. Picking the best path through the top 16 would raise the acceptance length from 4.27 to 6.79. Accuracy also falls toward the end of the block, which no selection among candidates can fix. Key idea. Two changes address the two problems. First, keep the top 16 candidates at each position, score every adjacent pair with a low-rank bilinear function, and walk the best-scoring path; the scoring is parallel and only the final walk is sequential. Second, insert a two-tap dynamic depthwise convolution before and after each attention and feed-forward sublayer, so that dependencies between neighboring positions do not have to pass through attention. Findings. Across benchmarks, DFlash 2 produces 16-25% more accepted output per verification pass than DFlash for about 1% added cycle latency. On the same Qwen3-4B GSM8K setup at temperature zero, the path selector alone reaches an acceptance length of 4.61, compared with 4.49 for the DSpark correction, while adding 2.0M parameters instead of 77.8M. Why it matters. Choosing among candidates the drafter has already produced costs far less than predicting a new distribution. It separates selection error, which a selector can fix, from the loss of accuracy toward the end of the block, which needs a change to the drafter itself. Adoption. SGLang, vLLM, and llama.cpp have merged DFlash 2 support. |
| DSpineDraft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding | 2026 |
Background. Domino and DSpark condition each draft position on its predecessors with a lightweight module that runs after the parallel backbone has finished. Problem. Predecessor information then enters only at the end of the network. An analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that an accurate predecessor helps its successor more when it enters earlier. Key idea. Inject the conditioning throughout the backbone. At every layer, a gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal chain unfolds over network depth while all positions still update in parallel. Features are passed in a space built from the target model's output embeddings, and layer-wise supervision in that space encourages shallow layers to form predictions. Findings. On Qwen3-8B at temperature zero, the mean acceptance length across seven math, code, and chat benchmarks rises from DFlash's 3.77 to 4.82, a 27.8% increase. In SGLang serving tests, DSpine delivers 23.3% higher throughput than DFlash on average. Why it matters. Where the conditioning enters the drafter matters as much as whether it is present. Spreading it across layers keeps the drafting cost of one parallel pass. |
| H-SpecParallel Speculative Decoding Without a Drafter-Side KV Cache | 2026 |
Background. Block drafters condition on the target model by projecting its hidden states at every input position into a separate KV cache that belongs to the drafter. Problem. That cache costs memory and KV writes for every request, and both grow with concurrency, which is the regime serving systems run in. Reusing the target's own KV cache in place removes the extra cache, but draft quality then fails to hold across the block. Key idea. Combine two sources of target context. Attention modules reuse the target's KV cache in place, and Mamba modules are initialized from the target's hidden state at the last input position only. The result is a hybrid Mamba-attention drafter with no drafter-side KV cache, and Mamba's parallel scan keeps the drafting block-parallel. Findings. Across three target models, H-Spec improves mean accepted length by 5.0-13.3% and batch-size-1 inter-token latency speedup by 5.3-12.6% over the best baseline. Under concurrent serving, it achieves higher throughput with lower KV cache utilization than the baselines at every evaluated concurrency level. Why it matters. A drafter's memory footprint is a serving cost that single-request benchmarks do not show. Once many requests share the GPU, a drafter that holds its own per-request KV cache takes memory that the scheduler would otherwise use for more requests. |
| UnoUnlocking Lossless Speedups in LLMs via Discrete Diffusion | 2026 |
Background. Speculative decoding keeps the target model's output exact but needs a separate drafter. Diffusion language models generate several tokens in parallel without a drafter but fall short of autoregressive models in quality. Problem. You would like the parallelism of diffusion and the quality of an autoregressive model in one model, with no second model to train, store, and schedule. Key idea. Give one model two sets of weights. The autoregressive weights, trained with next-token prediction, define the output distribution, and lightweight diffusion weights, learned in a distillation phase that adds negligible training overhead, generate several tokens at once. A family of samplers, Ψ-Spec, draws tokens in parallel from the autoregressive distribution, so the speedup is lossless. Findings. Uno achieves higher throughput than leading speculative decoding methods at every evaluated batch size, and up to 3× speedup over the base autoregressive model, including at the largest batch size the device supports. The 8B Uno model outperforms the 26B DiffusionGemma and the proprietary Mercury 2 on the evaluated agentic tool-use, coding, and long-context reasoning benchmarks. Why it matters. It moves the drafter from a separate model into the target model's own weights. Read it with DeepSeek-V3's MTP modules above, which also put the drafter inside the checkpoint but draft one token per module. Adoption. SGLang ships Uno's proposal-tree construction for speculative decoding. |
| Drafting from text already seen | ||
| RESTRetrieval-Based Speculative Decoding | 2023 |
Background. Every drafter to this point computes its guesses with a neural network: an independent small model, the target's own layers, or a head or block drafter attached to the target. Most of them pay for it in extra parameters, in training, or in both. Problem. Much of what a model emits is text it has effectively seen before: boilerplate, code idioms, and spans quoted straight out of the prompt. Spending a forward pass to predict those tokens is waste, and any learned drafter has to be trained again for each new target model. Key idea. Retrieve the draft instead of generating it. Match the current suffix against a corpus, take the continuations that follow the matches, assemble them into a candidate tree, and let the target model verify in the usual way. There are no draft parameters and no training, so one datastore serves any model. Findings. On 7B- and 13B-parameter models at batch size one, REST reports 1.62-2.36× speedups across code and text generation. Why it matters. It makes the drafter a data structure. The speedup then depends on how much of your workload is retrievable rather than on the quality of a second model, which is a useful thing to be able to reason about separately when traffic is repetitive. |
| lookahead decodingBreak the Sequential Dependency of LLM Inference Using Lookahead Decoding | 2024 |
Background. Autoregressive decoding is a sequential fixed-point problem, each token conditioned on the ones already emitted. Every speculative method breaks that sequence by adding a proposer: a draft model, a set of heads, or a datastore. Problem. Each of those proposers is something extra to build and keep. A draft model needs training and serving, heads need training against a specific target, and retrieval needs a corpus that resembles the workload. None of them is available at the moment you have only the model. Key idea. Guess several future tokens at once and refine them in place with Jacobi iteration, so a step both commits verified tokens and improves the outstanding guesses. The n-grams verified along the way go into a pool the next step draws candidates from. There is no draft model and no training at all. Findings. Lookahead decoding accelerates the evaluated MT-Bench workloads by up to 1.8× on one accelerator and reaches 4× speedup through strong scaling on multiple GPUs for code completion. Why it matters. It is the pure form of the question this topic asks: how much parallelism is available in decoding when you add nothing to the model. Whatever it recovers bounds what the extra machinery in every other method is actually buying you. |
| prompt lookupPrompt Lookup Decoding | 2023 |
Background. REST retrieves drafts from an external corpus that has to be built and searched. Lookahead decoding needs no corpus, but it spends extra positions in every forward pass to build its n-gram pool by Jacobi iteration. Problem. In summarization, document question answering, multi-turn chat, and code editing, the output repeats spans of the input: entity names, phrases, and code. The continuation is often already in the prompt, and no drafter is needed to find it. Key idea. Replace the draft model with string matching. Take the last few generated tokens, find an earlier occurrence of that n-gram in the prompt, and propose the tokens that followed it as the draft. The target model verifies them as usual, so the output is unchanged. Findings. With Mistral-7B-Instruct and greedy decoding on one A100 40GB, prompt lookup runs 2-4× faster than plain greedy decoding on summarization and context-based question answering. Why it matters. It sets the cost baseline for any drafter: a few lines of code, no weights, and no training. A learned drafter has to beat it on your workload by enough to pay for training and serving the drafter. Adoption. vLLM's n-gram proposer and the prompt-lookup option in Hugging Face Transformers' generate implement this method. |
| SuffixDecodingExtreme Speculative Decoding for Emerging AI Applications | 2024 |
Background. Prompt lookup matches against the current prompt only. Agent frameworks send many requests whose outputs repeat across calls: multi-agent pipelines run similar subtasks, and self-refinement loops rewrite the same output several times. Problem. Those workloads produce long, predictable sequences, but a fixed draft length cannot exploit them. A short draft leaves long accepted runs unused, and a long draft wastes verification when the match is wrong. Key idea. Cache token sequences from prompts and previous outputs in suffix trees, and draft by walking the tree from the current suffix. The draft length adapts to the match: speculate many tokens when the tree indicates acceptance is likely and few when it does not. Findings. On agentic benchmarks including SWE-Bench and Text-to-SQL, SuffixDecoding reaches speedups of up to 5.3×. It is 2.8× faster than EAGLE-2/3 and 1.9× faster than the model-free Token Recycling. Why it matters. The best drafter depends on the workload. When requests repeat each other, a data structure over past outputs predicts more than a neural drafter trained on general text. Adoption. vLLM ships a SuffixDecoding proposer. |
| speculative editsEditing Files at 1000 Tokens per Second | 2024 |
Background. Prompt lookup and SuffixDecoding find a draft by matching the latest output against text seen earlier. A code editor knows more than either: when it asks a model to apply an edit, the request already contains the current file, and an edit usually leaves most of that file unchanged. Problem. Cursor's apply step has the model rewrite the whole file instead of writing a diff, because models produce diffs less accurately: a diff leaves fewer output tokens to reason with, diff formats are rarer than whole files in training data, and models count line numbers poorly. The latency of a full rewrite, however, grows with the length of the file rather than with the size of the edit. Key idea. Use the current file as the source of draft tokens. A deterministic algorithm proposes the upcoming text from the file, so no draft model runs, and a Llama-3-70B model fine-tuned for full-file rewrites verifies each proposed chunk in one pass. Findings. The fine-tuned model rewrites files at about 1,000 tokens per second, about 13× faster than vanilla inference with Llama-3-70B and about 9× faster than Cursor's previous deployment of speculative edits on GPT-4. Why it matters. The best draft can come from the application rather than from a model. When most of the output is known in advance, acceptance is set by how much of the file survives the edit. A small change to a large file is mostly verified in long chunks, while a rewrite that changes most lines gains little over ordinary decoding. Adoption. OpenAI's Predicted Outputs API applies the same idea for any caller: a request passes the expected output, often the current file, as a prediction, matching tokens return faster, and prediction tokens that are rejected are billed at completion-token rates. |
| Verification trees and acceptance rules | ||
| SpecInferAccelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification | 2023 |
Background. The first speculative decoding papers described a decoding algorithm and measured it on single-sequence generation. A serving system has to run many requests concurrently, under a scheduler, with batching and a memory budget. Problem. A single draft sequence is discarded from its first disagreement onward, so one linear guess wastes most of the work it did. Verifying several alternative continuations at once would recover it, but the attention kernels and the scheduler both assume one linear sequence per request. Key idea. Let several small models propose a tree of candidate tokens, then verify the entire tree against the large model in one pass with a token-tree verifier that shares the prefix's attention work across branches. The tree, not the sequence, becomes the unit the serving system schedules and verifies. Findings. SpecInfer outperforms the evaluated serving baselines by 1.5-2.8× for distributed inference and 2.6-3.5× for offloaded inference while preserving generation quality. Why it matters. This is where speculation stops being a decoding trick and becomes a serving design. Tree verification and the branch-aware attention it requires are what later engines inherited, and the argument is made in a serving system's terms of throughput rather than one request's latency. |
| SequoiaScalable, Robust, and Hardware-aware Speculative Decoding | 2024 |
Background. Tree speculation hands the target model a set of candidate continuations and verifies them in one pass. The tree's shape, how wide it is, how deep it goes, and where it branches, is normally picked by hand or by a fixed heuristic. Problem. The best shape depends on three things at once: the acceptance rate at each depth, the sampling temperature, and how much parallel verification the hardware gives you for free. A tree tuned for one setting is wrong in the others, and one tuned for a given GPU is wrong on the next. Key idea. Treat the shape as an optimization problem. Given measured acceptance behavior, solve for the topology that maximizes expected accepted tokens per step, draw candidates so the result holds across temperatures, and size the tree from the target hardware's own compute and memory profile instead of a constant. Findings. On an A100, Sequoia accelerates Llama-2-7B by up to 4.04×, Llama-2-13B by 3.73×, and Vicuna-33B by 2.27×. In the evaluated L40 offloading setup, exact Llama-2-70B inference reaches 0.56 seconds per token, 9.96× faster than the paper's optimized baseline. Why it matters. It converts a tuning knob into something you compute, and it names the hardware term explicitly: the right amount of speculation is a property of the machine as much as of the model pair. That is the framing you need to move a speculative configuration between deployments. |
| SpecExecMassively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices | 2024 |
Background. Speculative decoding is normally analyzed with both models resident in GPU memory, where a target step costs one pass over weights already on the device. Consumer hardware cannot hold a large target model, so its weights are offloaded and streamed in per step. Problem. When weights stream from host memory, the cost of a target step is set by the transfer rather than by how many tokens that step verifies. A draft tree sized for a resident model leaves nearly all of that transfer unused. Key idea. Build a very large draft tree with the small resident draft model, then verify the whole tree in one offloaded target pass, so a single expensive step commits many tokens. The tree is sized to the step cost instead of to a fixed draft length. Findings. SpecExec generates as many as 20 tokens per target-model iteration. With RAM offloading on consumer GPUs, 50B-plus-parameter models reach 4-6 tokens per second at 4-bit precision and 2-3 tokens per second with 16-bit weights. Why it matters. Tree size is not a constant of the method. It follows from the ratio of target step cost to draft cost, and that ratio depends on where the weights live, so the datacenter setting and the consumer setting want opposite answers. |
| block verificationBlock Verification Accelerates Speculative Decoding | 2024 |
Background. Standard verification checks draft tokens one at a time, left to right. Each token is accepted or rejected by comparing the draft and target probabilities at its own position, and the first rejection ends the block. Problem. Token-by-token verification is one way to keep the output distribution exact, but it does not accept the most tokens possible. Deciding each position in isolation rejects some drafts that a rule considering the whole block could keep. Key idea. Verify the block jointly. The rule decides how much of the draft to accept from the probabilities of the whole block, still guarantees that the output is distributed exactly as the target model's samples, and is proven optimal in the expected number of tokens per iteration, so it is never worse than token-level verification. Findings. Block verification gives wall-clock speedups of 5-8% over standard token verification across the evaluated tasks and datasets. Why it matters. The two derivations at the top of this page fix the acceptance rule, and most later work leaves it alone. This entry shows the rule can still be improved without giving up exactness, and the change is confined to the verifier. Adoption. SGLang implements block verification as an option in its speculative rejection sampler. |
| Judge DecodingFaster Speculative Sampling Requires Going Beyond Model Alignment | 2025 |
Background. Every verification rule above preserves the target model's output distribution. Under greedy decoding a draft token is kept only if it equals the target's top choice, and under sampling it is kept with probability min(1, p/q), where p and q are the target's and the drafter's probabilities for that token. Acceptance therefore measures agreement with the target, not whether the draft is correct. Problem. Strong drafts are rejected early. With GPT-4o drafting a correct answer for Llama-3.1-405B, standard verification accepts about two tokens before the first rejection, and correct human-written text is also rejected early. A drafter aligned more closely with the target only moves its drafts toward the target's own wording, so a correct draft phrased differently is still rejected. Key idea. Keep a draft token if either standard verification or a learned judge accepts it. The judge is a linear head of about 16K parameters on the target's last-layer embedding, trained with the target frozen on about 500 hand-labeled examples, each a question with a correct and a wrong answer. Tokens before an error are positive examples and the erroneous token is negative. A threshold sets how lenient the judge is, and the strictest setting recovers standard speculative decoding. Findings. With Llama-3.1-8B drafting for Llama-3.1-405B, Judge Decoding runs about 9.7× faster than standard decoding in Hugging Face's implementation and 3.9× faster in the optimized gpt-fast framework, where it reaches 129 tokens per second on 8 H100 GPUs. Accuracy on GSM8K, HumanEval, MT-Bench, ARC, and MMLU stays close to the target model's. Why it matters. It separates agreement from correctness and gives up exactness to exploit the difference, so the output no longer follows the target's distribution and quality has to be checked on each task. The judge is also limited by its training data: removing coding examples from it lowered HumanEval accuracy from 86.6% to 80.4%, still above the drafter's 71.3%. The two entries below analyze what relaxing the acceptance rule costs. |
| lossy verificationRevisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes | 2026 |
Background. Judge Decoding, the entry above, gives up exactness to accept more tokens. Every other method above is lossless: its acceptance rule is constructed so that the tokens surviving verification are distributed exactly as the target model's own sampling would be. Problem. Relaxing that rule buys acceptance rate and quietly rewrites the output distribution. The speedup is easy to measure and the distributional cost is not, so a lossy variant can be reported as a strict improvement. Key idea. Classify lossy verification schemes into two families, truncation-based and collaborative, then analyze the distributions each family induces under a shared diagnostic framework. Findings. Truncation-based methods can perform substantially worse than the true truncation-sampling baseline because they distort its distribution. Collaborative methods avoid low-quality outputs only when the overshoot of draft probabilities over target probabilities remains controlled. Why it matters. The lesson generalizes past speculative decoding to any approximation sold as a speedup with an unstated distributional cost. It gives you two questions to ask: which family the scheme belongs to, and what its failure looks like. |
| relaxed acceptanceA Practical Investigation of Training-free Relaxed Speculative Decoding | 2026 |
Background. The lossy verification entry above classifies schemes that relax the acceptance rule and analyzes the distributions they induce. Papers proposing such schemes argue that relaxation gives further speedups, a controllable trade-off between capability and speed, or even capability gains. Problem. Those claims come from narrow and differing experimental settings, so the methods cannot be compared with each other or checked against current drafters. A lossless method needs no capability evaluation; a relaxed one does, and that cost is rarely counted. Key idea. Put the training-free relaxed methods in one framework and benchmark them on current drafter-verifier pairs and reasoning benchmarks. The comparison varies draft length, drafter cost, and relaxation parameters, and includes strict speculative decoding with a tuned draft length as a baseline. Findings. Tuning the draft length under the strict rule changes the speedup about as much as relaxation does, keeps the lossless guarantee, and needs no capability evaluation. Relaxing with a drafter that is a weak standalone language model, including a dedicated MTP drafter, gives poor capability-speed trade-offs, and relaxation hyperparameters tuned on one task fail to transfer to others. Why it matters. It gives you an order of operations: tune the lossless configuration first, then relax only with a strong drafter and an evaluation on every task you deploy for. Drafters are moving toward small dedicated MTP modules, which is the case where relaxation helps least. |
| Inside a serving system | ||
| vLLM anatomyInside vLLM: Anatomy of a High-Throughput LLM Inference System | 2025 |
Background. The papers on this page treat speculation as an algorithm or a policy. In a serving engine it is one more phase of a step loop that already runs continuous batching, chunked prefill, prefix caching, and paged KV allocation for every running request. Problem. How speculation fits into that loop determines what it costs. Draft tokens need KV blocks before they are verified, one forward pass must carry requests with different numbers of new tokens, and the sampler must apply the acceptance rule to each request. Papers rarely show this level of detail, and the engine's source spreads it across many modules. Key idea. Follow vLLM V1, at one commit, from a single request to a multi-GPU deployment: the engine core and scheduler, the paged KV cache manager, chunked prefill, prefix caching, guided decoding, speculative decoding, disaggregated prefill and decode, and data-parallel scaling, ending with the latency metrics and roofline model used to benchmark it. For speculation, the engine proposes k draft tokens with an n-gram, EAGLE, or Medusa proposer after a step samples its output, the next step's scheduler reserves KV blocks for them, and a rejection sampler replaces ordinary sampling after the target's forward pass. Why it matters. It shows where the costs of speculation land in a real engine. Each step's draft depends on the tokens the previous step accepted, so drafting sits between verify passes on the critical path of every step, and draft tokens take KV blocks and batch slots that could otherwise admit more requests. Most of the serving entries below decide how much of that capacity speculation should get. |
| SmartSpec / TurboSpecClosed-loop Speculation Control System for Optimizing LLM Serving Goodput | 2024 |
Background. A serving engine batches requests continuously, and speculation adds k draft tokens per request to each verify pass. The first version of this paper called the system SmartSpec, under the title Optimizing Speculative Decoding for Serving Large Language Models Using Goodput; later versions call it TurboSpec. Problem. No fixed draft length suits every load. When the request rate is low, the batch leaves compute idle and verification costs little, but at high request rates or low acceptance, verifying rejected drafts takes compute that other requests need, and speculation raises latency instead of lowering it. Key idea. Choose the draft length k for the batch at every step to maximize goodput, the number of tokens that reach the output divided by the step's execution time. The expected tokens per request follow from an acceptance rate tracked as a moving average over earlier steps, and the step time comes from a linear model of batched and context tokens fitted by an offline profiler. At high load the best k is 0, so speculation turns itself off. Findings. Implemented in vLLM and evaluated with draft models, EAGLE, and prompt lookup on Llama-family targets, TurboSpec reduces latency by up to 3.16× relative to non-speculative decoding. Why it matters. It turns the draft length from a deployment constant into a scheduling decision made every step from quantities the engine already measures. Goodput, not acceptance rate or draft length, is the objective, because a longer accepted run is worth nothing if the step that produced it took time other requests needed. |
| MagicDecBreaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding | 2024 |
Background. The standard case for speculative decoding is that decode at small batch is memory-bound, so verifying several tokens costs barely more than generating one. The corollary usually drawn is that speculation buys latency at low batch and costs throughput at high batch. Problem. At long context and large batch the usual assumptions invert. KV cache traffic rather than weight loading dominates the target step, and it grows with both batch size and sequence length, so the reasoning that decides when speculation pays no longer applies. Key idea. Work out which term actually dominates at a given batch size and sequence length, then hold the draft to a fixed-size sliding-window KV cache so drafting stays cheap as context grows. Speculation then improves latency and throughput together in the long-context regime. Findings. For moderate-to-long sequences and batch sizes from 32 to 256, MagicDec accelerates Llama-3.1-8B by up to 2.51× across the evaluated hardware and tasks. Why it matters. A tradeoff presented as fundamental can be an artifact of an operating point. This is the entry that forces you to ask which term dominates before reusing a rule of thumb about memory-bound decode. |
| D-cutAdaptive Verification Depth Pruning for Batched Speculative Decoding | 2026 |
Background. Parallel drafters decouple draft length from drafting latency, so they propose long drafts and raise the mean number of accepted tokens for a single request. The cost of verifying a long draft is paid for every request in the batch. Problem. Under high concurrency, verifying long drafts spends much of the batch's compute on tokens that will be rejected. Speculative decoding can then run slower than plain autoregressive decoding. Key idea. Prune drafts jointly across the batch. Acceptance lengths differ between concurrent requests, so the verification budget goes to the draft tokens with the highest confidence, whichever request they belong to. A runtime cost model, which accounts for the GPU architecture and the parallelism strategy, sets how deep to prune. Findings. Under high concurrency on dense and mixture-of-experts models, D-cut raises the average speedup from 1.26× to 1.65×. It restores a speedup in dense-model configurations where long-draft baselines run slower than autoregressive decoding, and it reaches up to 3.0× over autoregressive decoding on mixture-of-experts models. Why it matters. The right draft length depends on the batch, not only on the request. Read it with MagicDec above: both ask which cost dominates at the batch size you actually run. |
| CascadeUtility-Driven Speculative Decoding for Mixture-of-Experts | 2025 |
Background. Verification is nearly free in a dense model because every token in the verify pass uses the same weights, which the step reads once. In a mixture-of-experts model each token is routed to a few experts, so a decode step reads only the experts its tokens select. Problem. Draft tokens are routed to different experts, so verifying them reads more expert weights than decoding a single token, and verification time rises 2-3×. Always-on speculation then slows decoding by up to 1.5× relative to no speculation, and the effect depends on the task: with Mixtral and k = 3, speculation is 39% faster on code and 54% slower on math. Key idea. Measure the utility of speculation, the ratio of token gains to verification cost, while each request runs. A test phase tries up to four values of k for a few iterations each, and a set phase runs the best one for 16 iterations. If utility stays below 1 even at k = 1, speculation is turned off, and each consecutive off phase lasts twice as long before the next test. The drafter keeps running while speculation is off so that its KV cache stays current, which costs 2-3% on average. Findings. Implemented in vLLM and evaluated at batch size one on an RTX 6000 Ada GPU with five MoE models, including Mixtral 8×7B, OLMoE, and Qwen1.5-MoE, Cascade limits the worst-case slowdown relative to no speculation to 5% and raises throughput by 7-14% over static choices of k. Why it matters. The argument that verification is free assumes every verified token uses the same weights. Expert routing breaks that assumption, so whether to speculate must be measured per request and rechecked as the request runs. Utility plays the role of SmartSpec's goodput but is measured by short trials rather than predicted by a cost model, which suits a verification cost that depends on which experts the drafts select. |
| mirror speculative decodingBreaking the Serial Barrier in LLM Inference | 2025 |
Background. Draft-then-verify runs in strict alternation. The draft model produces candidate tokens, the target model verifies them, and only then does the draft resume from whatever was accepted. Problem. Alternation means one of the two models is idle at every instant, and the latency of a round is the sum of both phases. On a deployment with more than one kind of accelerator, whichever device is not in its phase does nothing. Key idea. Run both directions at once. The draft proposes continuations while the target proposes correction paths for the draft, with the two halves placed on heterogeneous accelerators. The tradeoff is a deployment that needs both a GPU and an NPU plus a tight cross-device synchronization budget. Findings. On SpecBench with 14B- to 66B-parameter models, Mirror-SD reports 2.8-5.8× wall-clock speedups and a 30% average relative improvement over EAGLE-3. Why it matters. It reframes speculation as a placement and pipelining problem across unlike devices rather than an algorithm running on one. Given heterogeneous silicon, the serial loop is leaving a whole device idle, and that idleness is the thing to attack. |
| online spec decodingOnline Speculative Decoding | 2023 |
Background. A draft model is trained once, offline, on a general corpus, and then shipped with the serving stack. The speedup it delivers is governed by its acceptance rate against the target model. Problem. Acceptance rate depends on how closely the draft agrees with the target on the text actually being generated, and a deployed model sees a narrow query distribution that shifts over time. A general draft is mismatched where the traffic is, and the mismatch costs speed silently, as rejected tokens rather than wrong output. Key idea. Keep updating the draft model during serving on the live query distribution, taking the target's outputs on the queries that arrive as the training signal. The draft becomes a component that tracks traffic instead of a fixed artifact shipped beside it. Findings. Across synthetic and real query streams, online adaptation raises token acceptance by 0.10-0.65 and reduces latency by 1.42-2.17× relative to the paper's offline-draft baseline. Why it matters. Acceptance rate turns into something you maintain rather than a number measured once at deployment. It also puts a training loop inside a serving system, which raises the question of what capacity pays for the updates and when. |
| AdaServeAccelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding | 2025 |
Background. A serving engine applies one speculation setting to the whole running batch, on or off, with a single draft length or tree shape. The requests sharing that batch arrive with different latency targets. Problem. One setting cannot satisfy several SLOs at once. A tree sized for the tightest target spends compute the looser requests would rather have as throughput, and a tree sized for throughput misses the tight target. Key idea. Make per-request speculation a scheduling variable. A constrained-optimization formulation builds a speculation tree sized to each request’s latency target, and a speculate-select-verify pipeline trades decode speed against throughput within one shared batch. Findings. Across the evaluated workloads, AdaServe reduces SLO violations by up to 4.3× and improves goodput by up to 1.9× over the strongest baseline. Why it matters. This is the serving-system view of speculative decoding once SLOs stop being uniform. Speculation stops being a deployment flag and becomes a per-request decision, made by the scheduler with the same information it uses to batch. |
| AgentSpecSpeculative Decoding for Batch Inference of LLM Agents | 2026 |
Background. Speculation pays when a decode step is memory-bound and has arithmetic to spare for checking extra tokens. That spare capacity shrinks as the batch grows, and agent serving runs at large batch, with many trajectories in flight at once. Problem. At those batch sizes speculative decoding loses most of its speedup. The two causes identified here are a high rejection rate for drafted tokens and a dynamic token budget that goes unused. Key idea. Constrain drafting to semantically coherent segments of the agent workflow, which cuts speculation down irrelevant paths, and allocate the free token budget using agent-level information. Why it matters. It recovers a lost optimization from structure the serving layer normally discards, namely what the agent is doing next. Read it with the agent-serving meetings, where the batch sizes come from. |
| Training drafters | ||
| SpecForgeA Flexible and Efficient Open-Source Training Framework for Speculative Decoding | 2026 |
Background. EAGLE-3 drafting is in every major engine, and the speedup depends on a draft head trained against the specific target model. Trained heads exist for a handful of popular open checkpoints. Problem. The algorithm is settled and the drafts are not. A newly released base model has no head, and producing one at production quality means training a draft against a large target, which is a distributed-training problem rather than an inference one. Key idea. Build a draft-training framework with target-draft decoupling, hybrid parallelism, and tuned kernels. SpecBundle adds released EAGLE-3 drafts for mainstream open models. Findings. For Qwen3-235B-A22B, SpecForge trains EAGLE-3 drafts up to 9.9× faster than the evaluated training baseline. The released SpecBundle drafts deliver up to 4.48× end-to-end inference speedup in SGLang. Why it matters. The practical blocker it addresses is that good draft models, not the algorithm, are what the community is missing. Whether an inference optimization is available to you often turns on training infrastructure and released artifacts rather than on the published method. |
| SpeculatorsA unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM | 2025 |
Background. Drafters are published from many research repositories, each with its own checkpoint layout, configuration, and training script. A serving engine has to load each of them, and a user who wants a drafter for a new target model has to find a recipe whose output the engine accepts. Problem. Without a shared format, training a drafter and serving it are coupled through one-off conversion code. Every new drafter algorithm needs both a training implementation and a loader, and the two can drift apart. Key idea. Define one Hugging Face-compatible format for draft models and build the training pipeline around vLLM. vLLM generates the target model's hidden states, either offline to disk or online across nodes; the library trains drafters on them for algorithms including EAGLE-3, DFlash, and DSpark; and the resulting checkpoints load in vLLM directly. Converters bring checkpoints from external research repositories into the same format. Why it matters. A shared checkpoint format separates who trains a drafter from who serves it. A new drafter algorithm then needs one training implementation and one loader, and any checkpoint in the format runs in the engine. |
| TorchSpecA PyTorch native library for training speculative decoding models | 2026 |
Background. Training a drafter requires the target model's hidden states for every training token. The target is the large model, so producing those states is an inference workload of its own, separate from training the small drafter. Problem. Writing hidden states to disk first costs storage proportional to the number of training tokens, the hidden size, and the number of captured layers; fal estimated about 38 TB for its 250K-sample training set. Running the target and the trainer on the same GPUs instead ties their scaling together. Key idea. Disaggregate the two. Inference engine groups run the target model and stream its hidden states through the Mooncake store directly to distributed training workers, without writing them to disk, so each side scales independently. Why it matters. Draft training for a large target joins a serving system to a training system. Treating hidden-state generation as an inference job lets existing serving engines do the expensive half. Adoption. DigitalOcean, CoreWeave, and fal report training production drafters with TorchSpec: an EAGLE-3 drafter for MiniMax-M2.5, a DFlash drafter for Kimi K2.7 Code, and a DSpark drafter for a prompt-expansion model, respectively. |
| D-PACEDynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting | 2026 |
Background. A parallel drafter such as DFlash predicts the whole block in one pass and is trained with cross-entropy at every block position. The positions are weighted by a fixed schedule; DFlash decays the weight exponentially along the block. Problem. Which positions limit the accepted length changes as the drafter trains. A fixed schedule keeps spending training signal on positions that are already accurate and under-weights the position where drafts are currently cut off. Key idea. Derive the weights from the quantity that sets the speedup. A differentiable surrogate of expected accepted draft length gives each position a weight matched to its contribution to that surrogate's gradient, so the weights move to whichever positions currently limit acceptance. Smoothing is applied only when computing the weights, which keeps training stable without changing the cross-entropy term. Findings. On Qwen3-4B DFlash drafters, D-PACE improves average decoding speedup by 8.0-9.7% and average emitted length by 8.5-10.7% over DFlash's fixed schedule. Emitted length rises by about 12.5% on Llama-3.1-8B-Instruct and 12.8% on Qwen3-8B, at 2.3% training-time overhead and with no change to the drafter architecture or the inference procedure. Why it matters. Draft quality depends on the training objective as well as the architecture. Tying the loss to accepted length raises acceptance with no change at inference time, so an engine that already serves a drafter benefits once the drafter is retrained. Adoption. SpecForge, NVIDIA Model Optimizer, TorchSpec, and Speculators implement D-PACE as a training loss for parallel drafters. CoreWeave trained the DFlash drafter for its Kimi K2.7 Code endpoint with D-PACE and reports about an 8% improvement in acceptance length. |