Skip to content

LLaDA2.0: Scaling Up Diffusion Language Models to 100B

ArXiv: 2512.15745

🎯 Pitch

LLaDA2.0 presents a practical recipe to scale discrete masked diffusion language models to 100B parameters by systematically converting pretrained auto-regressive checkpoints using a novel Warmup–Stable–Decay continual pre-training schedule and supporting techniques (document-level attention masks, complementary masking, confidence-aware training, ELBO-based DPO). This enables large-scale diffusion models that inherit AR knowledge while unlocking parallel, bidirectional generation—reducing inference latency and improving performance on structured tasks like code and math, making diffusion LLMs viable for frontier-scale deployment.

1. Executive Summary (2-3 sentences)

LLaDA2.0 describes a practical way to scale discrete masked diffusion language models (dLLMs / MDLMs) up to a 100B-parameter frontier model by converting an existing strong auto-regressive (AR) checkpoint instead of training a diffusion model from scratch. The core contribution is a 3-phase continual pre-training schedule—Warmup–Stable–Decay (WSD)—plus training/inference modifications (document-level attention masking, complementary masking in SFT, confidence-aware training, and ELBO-based DPO) that make the AR→diffusion transition stable and efficient while retaining the parallel-decoding benefits of diffusion.

2. Context and Motivation

  • Problem / gap
  • AR LLMs predict the next token left-to-right, which makes decoding inherently sequential and hard to parallelize at inference time (Introduction).
  • Masked Diffusion Language Models (MDLMs) generate by iterative denoising from masked text using bidirectional context, which can enable more parallel generation, but existing diffusion LMs are mostly small-scale (the text highlights ≤8B for many diffusion works, Introduction + Related Work §2.1–2.2).

  • A major open question is whether diffusion LMs can be brought to hundreds of billions of parameters with practical training and infrastructure constraints (Related Work §2.2).

  • Why it matters

  • If diffusion LMs can scale while preserving strong capability, they could reduce inference latency bottlenecks via parallel decoding and potentially help tasks that benefit from bidirectional / “holistic” context (Introduction).

  • Prior approaches and shortcomings (within the provided content)

  • Training MDLMs from scratch can reach competitiveness at ~8B (Related Work §2.1), but from-scratch diffusion remains behind leading AR models and is expensive, limiting scale.

  • AR-initialized diffusion approaches exist (Related Work §2.2), including immediate conversion or mask annealing, and BDLM (block diffusion) hybrids that interpolate AR and diffusion; however:

    • They are reported at smaller scales (7B–30B), and
    • BDLM training efficiency is a bottleneck when using large corpora (Related Work §2.2; also reiterated in Introduction).

  • How this work positions itself

  • It frames the key idea as systematic conversion of a pretrained AR model into a large-scale diffusion model using a staged training pipeline (Figure 2; §3–§5), aiming to preserve “knowledge inheritance” while gaining diffusion’s parallel decoding.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

  • The system is a training + inference recipe that turns a pretrained AR transformer into a diffusion-style language model that reconstructs masked tokens using bidirectional context.
  • It solves the AR-to-diffusion mismatch using a staged continual-pretraining schedule (WSD) and then post-trains the resulting model into an instruction-following assistant (SFT + DPO) while tuning it for efficient parallel decoding.

3.2 Big-picture architecture (diagram in words)

  • Input: a pretrained AR base model checkpoint (e.g., Ling-mini-2.0-base, Ling-flash-2.0-base, Figure 2 / §4.1).
  • Stage 1 (CPT): convert AR→diffusion with Warmup–Stable–Decay (WSD) using a document-level attention mask to prevent cross-document interference (§4.1–§4.2; Figure 2; Eq. (3)–(4)).
  • Stage 2 (Post-training):
  • SFT (block diffusion): instruction tuning using a block-diffusion reconstruction loss conditioned on prompt c (Eq. (5)), plus training tricks: mask ratio bandwidth + complementary masking (§5.1).
  • Optional CAP training: add an auxiliary confidence loss to sharpen correct predictions and improve parallel decoding efficiency (§5.2; Eq. (6); Figure 3).
  • DPO: preference alignment adapted to diffusion by using an ELBO-like reconstruction score (Eq. (7)–(8)).
  • Inference: block-wise diffusion sampling with a threshold-based acceptance rule and fallback to ensure progress (§5.4), with specific decoding hyperparameters used in evaluation (§6.1) and efficiency numbers (Figure 3; §7.3).

3.3 Roadmap for the deep dive

  • Explain the modeling objective difference between AR vs. masked diffusion (why conversion is hard).
  • Define BDLM and how “block size” connects AR, block diffusion, and full-sequence MDLM (Eq. (1)–(2)).
  • Walk through the WSD schedule and why each phase exists (Warmup→Stable→Decay, §4.1).
  • Detail the document-level block diffusion attention mask and what it prevents (Eq. (3)–(4), §4.2).
  • Cover post-training: SFT (Eq. (5)), complementary masking, mask bandwidth, then CAP (Eq. (6)) and DPO via ELBO (Eq. (7)–(8)).
  • Finish with inference mechanics + infrastructure choices and their measured speed/quality trade-offs (Figures 3–6; §6.3; §7).

3.4 Detailed, sentence-based technical breakdown

This is an empirical systems + training-recipe paper whose core idea is to continually pre-train an existing AR model into a masked diffusion model using a staged block-size schedule and then align it for instruction following and fast inference.

3.4.1 Core generative objective: AR vs. MDLM vs. BDLM

  • An AR language model generates text by predicting the next token given all previous tokens (left-to-right causal dependency, Introduction). This makes decoding sequential.
  • An MDLM (masked diffusion LM in this text) learns to reconstruct original tokens x0 from a corrupted version xt where some tokens are replaced by [MASK] (Introduction; Eq. (2)). Because the model can see unmasked tokens on both sides, it uses bidirectional context.
  • A BDLM (block diffusion LM) interpolates between AR and full-sequence diffusion by dividing a sequence into blocks of size LB and reconstructing masked tokens within the current block while conditioning on earlier clean blocks (Related Work §2.2; Eq. (1)).
  • The paper explicitly treats an AR model as a special case of BDLM with LB = 1 (one-token blocks, §4.1).
  • When LB equals the sequence length (they use LB = 4096), BDLM becomes equivalent to full-sequence MDLM (K=1 in Eq. (2), §4.1).

3.4.2 What happens first/second/third: end-to-end pipeline (Figure 2)

  1. Start from an AR base checkpoint (Ling-mini-2.0-base / Ling-flash-2.0-base, Figure 2; §4.1).
  2. Stage 1: Continual Pre-Training (CPT) to get an MDLM-like model
  3. Apply the WSD schedule that changes how the model is trained (block size and attention masking), so the model gradually shifts from AR-style dependencies to diffusion-style denoising (§4.1).
  4. Apply a document-level attention mask during training to prevent the model from attending across unrelated documents when sequences are packed for throughput (§4.2).
  5. After training, optionally apply top-k checkpoint merging by averaging parameters of the best k checkpoints (k not specified in the provided excerpt) to improve generalization (§4.3).
  6. Stage 2: Post-training for deployment
  7. Perform instruction SFT with a block diffusion objective conditioned on prompt c (Eq. (5)), using data-efficiency tricks (mask ratio bandwidth; complementary masking) (§5.1).
  8. Optionally run Confidence-Aware Parallel (CAP) training by adding an entropy-reducing confidence loss on correctly predicted tokens (Eq. (6); §5.2).
  9. Run DPO for preference alignment by replacing intractable diffusion log-likelihood with an ELBO-like reconstruction score over masked tokens (Eq. (7)–(8); §5.3).
  10. Inference
  11. Generate text block by block, and within each block perform iterative denoising with a threshold-based acceptance scheme and fallback (§5.4).
  12. Use inference hyperparameters (in evaluation): temperature = 0.0, block size = 32, decoding threshold = 0.95 (§6.1).

3.4.3 Warmup–Stable–Decay (WSD): why block size scheduling helps (Section 4.1)

The conversion challenge is described as a distribution / objective mismatch: AR training uses left-to-right prediction, while diffusion uses bidirectional denoising, and switching objectives abruptly can cause instability and “degradation of pretrained knowledge” (§4).

  • Warmup phase (progressive block size increase)
  • Start with LB = 1 (AR-like training view) and gradually increase block size through a schedule LB = 1 → 4 → 32 → 64 → 4096 (§4.1).
  • Increasing LB expands the receptive field within which the model jointly reconstructs tokens, nudging it toward bidirectional denoising without abruptly removing AR priors.
  • They require the sequence length be divisible by the block size to avoid fragmented blocks (§4.1).
  • Stable phase (large-scale full-sequence MDLM training)
  • Fix LB = 4096, effectively making the whole sequence one block (K=1), and train at scale under the MDLM objective (Eq. (2); §4.1).
  • The excerpt claims that at this point “the clean part of the attention computation … no longer needs to be maintained,” reducing attention compute and improving data processing efficiency (§4.1; see Figure 2 narrative).
  • Decay phase (reduce block size back for efficient inference)
  • Reduce block size from 4096 down to a smaller value (example given: LB = 32) step-by-step rather than abruptly (§4.1).
  • The intent is to distill global denoising ability learned in Stable into a compact block structure that supports KV-cache reuse and “fast variable-length generation” (benefits attributed to BDLMs; §4.1; Related Work §2.2).

Key point: WSD is a curriculum over the training mask/conditioning structure, not just over data. It connects AR-like learning to MDLM learning smoothly.

3.4.4 The BDLM/MDLM training losses (Eq. (1)–(2)) explained in plain language

  • The model sees:
  • A clean sequence x0.
  • A corrupted version xt where each token is masked with probability (1 − αt) for a diffusion “timestep” t.
  • BDLM loss (Warmup/Decay, Eq. (1))
  • The sequence is partitioned into K = Ltotal / LB blocks of length LB.
  • For each block k, the model predicts the original tokens only at masked positions inside that block (indicator 1[x^i_{t,k} = [MASK]]).
  • Predictions are conditioned on:
    • x0,<k: earlier blocks in clean form (AR-like prefix of clean blocks),
    • and xt,k: the noisy current block being denoised.
  • A time-dependent weight −α′t / (1 − αt) scales the contribution for the chosen timestep t.
  • MDLM loss (Stable, Eq. (2))
  • Equivalent to K=1 (one block), so the model predicts masked tokens in the full sequence conditioned only on xt (bidirectional within-document attention; Eq. (4) in MDLM case).

Micro-example (toy walk-through consistent with Eq. (1)/(5)): - Suppose the response tokens are: x0 = ["The", "cat", "sat", "on", "the", "mat", "."]. - Choose a block size LB = 3, giving blocks: - Block 1: ["The","cat","sat"] - Block 2: ["on","the","mat"] - Block 3: ["."] (in practice they pad/quantize lengths to multiples of LB, §5.1). - Corrupt Block 2 into xt,2 = ["on", "[MASK]", "mat"]. - The BDLM objective trains the model to predict the masked token "the" inside Block 2 conditioned on: - earlier clean blocks x0,<2 = ["The","cat","sat"] and - the noisy current block xt,2. - This preserves an AR-like “prefix is clean” structure across blocks while allowing diffusion-style denoising within the current block.

3.4.5 Document-level attention mask: preventing cross-document contamination (Section 4.2, Eq. (3)–(4), Figure 2)

Because they pack multiple documents into fixed-length segments for throughput, naive full attention would let tokens attend across unrelated document boundaries, which is harmful for bidirectional denoising (§4.2).

  • They define a document-level attention mask so attention operates only within the same document segment (Eq. (4) for MDLM case).
  • For block diffusion training they build a structured mask over a concatenated sequence x_full of length 2L that contains the noisy sequence xt followed by the clean sequence x0 (Eq. (3); Figure 2 description).
  • The mask M ∈ {0,1}^{2L×2L} uses block indices b(k) = ⌊k/LB⌋ and has three main components (Eq. (3)):
  • Noisy-to-noisy (xt→xt): block-diagonal attention within each noisy block (1_{b(i)=b(j)}), matching the “denoise within block” idea.
  • Noisy-to-clean (xt→x0): xt can attend to earlier clean blocks only (1_{b(i) > b(j−L)}), injecting AR-like conditioning across blocks.
  • Clean-to-clean (x0→x0): block-causal attention so a clean block can attend to itself and earlier clean blocks (1_{b(i−L) ≥ b(j−L)}).
  • Clean-to-noisy is disallowed (explicitly zero “prevent attention from queries in x0 to keys in xt,” §4.2).
  • The combination is described in Figure 2 as a mix of MBD (block diagonal), MOBC (offset block-causal), and MBC (block-causal), bounded by document boundaries.

Mechanistic effect: packed training improves throughput, and the document-level mask prevents the model from learning spurious cross-document dependencies that would otherwise destabilize bidirectional reconstruction (§4.2).

3.4.6 Post-training: SFT, complementary masking, mask ratio bandwidth (Section 5.1, Eq. (5))

  • SFT objective (Eq. (5))
  • For instruction tuning, the model conditions on a prompt c and reconstructs the response x0 from a masked version xt.
  • The conditioning resembles BDLM: within a block, predict masked tokens conditioned on prompt c, previous clean blocks x0,<k, and current noisy block xt,k.
  • Padding / length quantization
  • They round each sequence length up to a multiple of the block size so block boundaries align with the attention mask (§5.1).
  • Mask ratio bandwidth
  • Instead of sampling mask rates over the full interval (written as αt ∼ U[0,1]), they clip to [αmin, αmax] to avoid uninformative extremes (near-zero masking is trivial; near-total masking degenerates to marginal modeling) (§5.1).
  • The excerpt does not provide numeric values for αmin/αmax.
  • Complementary masking
  • For each training sequence x0, they create two masked versions: xt and x′t using the inverse mask (so positions masked in one are unmasked in the other) (§5.1).
  • Training on both ensures “near-100% data utilization” in the sense that every token is unmasked exactly once across the pair (§5.1).
  • Footnote clarifies they tried complementary masking in CPT and it only helps when corpus size is < 100B tokens; thus they keep it in post-training (§5.2 footnote).

3.4.7 CAP training: sharpening probabilities to unlock faster parallel decoding (Section 5.2, Eq. (6), Figure 3)

  • The issue: parallel decoding benefits can be limited if the model’s token distributions are not confident enough to accept many tokens per denoising step (§5.2).
  • Confidence-Aware Parallel (CAP) Training
  • Add an auxiliary loss L_conf that reduces entropy (makes the distribution “sharper”) only on tokens that are already correctly predicted in that step (§5.2).
  • Final objective: L(θ) = L_SFT(θ) + λ L_conf(θ) (Eq. (6)), with λ a hyperparameter (numeric value not given).
  • Claimed empirical effect (Figure 3)
  • Figure 3 reports better decoding efficiency for LLaDA2.0-flash with CAP (details below in §5 “Experimental Analysis” and §7 “Infrastructure”).

3.4.8 DPO adaptation to diffusion via ELBO (Section 5.3, Eq. (7)–(8))

Standard DPO for AR models depends on tractable log-likelihoods, which the excerpt describes as “intractable” for their diffusion policy (because the diffusion generation likelihood is not directly computed) (§5.3).

  • They define a block diffusion ELBO-like score B_BDLM(θ, x | c) that mirrors the reconstruction objective (Eq. (7)).
  • It is estimated via a single Monte Carlo sample over timesteps/noise (Et,xt[...] in Eq. (7)).
  • It sums log-probabilities only over masked tokens, weighted by the diffusion time weight α′t/(1−αt).
  • For a preference pair (x_w, x_l) (preferred vs dispreferred), the DPO loss (Eq. (8)) uses:
  • ∆B(x|c) = B_BDLM(θ, x|c) − B_BDLM(θ_ref, x|c), i.e., the policy’s ELBO advantage over a frozen reference model initialized from the post-SFT model.
  • β = 0.1 is explicitly given (§5.3) as the strength of deviation from the reference.

3.4.9 Inference: block-wise diffusion sampling with threshold acceptance (Section 5.4; plus evaluation settings §6.1 and tuning §6.3)

  • The model generates one block at a time, conditioned on prompt c and previously generated blocks: pθ(x^b_s | c, x^{<b}_t) (§5.4).
  • Inside a block, generation is iterative:
  • Sample candidate tokens for remaining masked positions.
  • Accept tokens whose probability exceeds a confidence threshold.
  • If too few tokens exceed the threshold, fallback to accepting a fixed number of the most probable tokens even if below threshold (§5.4).
  • In evaluation, they use block size = 32 and threshold = 0.95 for all LLaDA2.0 models (§6.1).

3.4.10 Infrastructure and stability mechanisms (Section 7; Figure 6)

  • Pretraining backend: Megatron-LM with multiple parallelism dimensions: DP, PP, TP, CP, EP (Figure 6; §7.1).
  • They ensure consistent masking by generating masks on a single model-parallel rank and broadcasting within MP ranks (§7.1).
  • Efficient attention implementation:
  • Use cuDNN attention to support arbitrary block diffusion attention masks, claiming >1.3× end-to-end speedup and >90% memory savings in the attention layer vs unfused TransformerEngine when training LLaDA2.0-mini (§7.1).
  • Apply “zig-zag partitioning” of the block diffusion attention mask for load balancing across the CP group (§7.1).
  • Numerical stability when switching AR→diffusion:
  • Problem described: masked token embeddings are unused in AR training and decay toward zero, which can cause gradient explosion at high mask ratios (§7.1).
  • Their mitigation is to add independent Gaussian noise to the embedding-layer output for each masked token during initial iterations to keep embedding norms significant without reinitializing embeddings (which could cause forgetting) (§7.1).
  • Post-training infra: dFactory on VeOmni with DP + EP, plus data packing like CPT (§7.2).
  • Inference engine: adapt dInfer for block diffusion inference and integrate support into SGLang to reuse AR-style optimizations like KV-cache reuse (§7.3).

Missing details (important for reproducibility): the provided excerpt does not specify optimizer, learning rates (except DPO LR initialized to final SFT LR), batch sizes, number of layers/hidden size/heads, tokenizer, total training tokens, compute budget, or hardware type/count.

4. Key Insights and Innovations

  • (1) Warmup–Stable–Decay (WSD) conversion schedule (Section 4.1; Figure 2)
  • Novelty: uses block size as a curriculum (LB=1→…→4096→…→32) to bridge AR causal dependencies to diffusion denoising without abrupt objective switching.

  • Significance: targets stability + data efficiency in AR→diffusion continual pre-training, enabling scaling to the 100B model regime described in the abstract.

  • (2) Document-level block diffusion attention mask for packed training (Section 4.2; Eq. (3)–(4); Figure 2)

  • Novelty: explicitly prevents attention across document boundaries during bidirectional denoising while still enabling a vectorized block diffusion forward pass over concatenated [x_noisy, x_clean].

  • Significance: addresses a concrete failure mode—spurious cross-document dependencies—claimed to cause semantic confusion/instability in bidirectional training (§4.2).

  • (3) Post-training data-efficiency tricks tailored to diffusion objectives (Section 5.1)

  • Complementary masking ensures every token contributes learning signal across paired masks, addressing the “partial learning signal” issue inherent to random masking.

  • Mask ratio bandwidth avoids extreme mask rates that create high-variance/low-signal gradients (§5.1).

  • (4) Confidence-Aware Parallel (CAP) training to improve practical parallel decoding (Section 5.2; Eq. (6); Figure 3)

  • Novelty: explicitly trains the model to become more confident (lower entropy) on already-correct tokens so that threshold-based parallel decoding can accept more tokens per forward pass.

  • Significance: directly targets throughput, not just benchmark accuracy.

  • (5) DPO reformulated over diffusion reconstruction ELBO (Section 5.3; Eq. (7)–(8))

  • Novelty: replaces AR log-likelihood terms with an ELBO-like masked-token reconstruction score, enabling preference optimization with diffusion policies.
  • Significance: provides a concrete alignment recipe at scale, including a stated preference dataset size of 1.5M pairs (§5.3).

5. Experimental Analysis

5.1 Evaluation methodology (Section 6.1)

  • Benchmark suite: 47 benchmarks across five categories (§6.1):
  • Knowledge (e.g., MMLU, MMLU-Pro, GPQA-Diamond, ARC, CMMLU, C-Eval, TriviaQA, …)
  • Reasoning (e.g., HellaSwag, BBH, DROP, SQuAD 2.0, …)
  • Coding (e.g., HumanEval, MBPP, LiveCodeBench, Spider, …)
  • Math (e.g., GSM8K, MATH, AIME 2025, …)
  • Agent & Alignment (e.g., BFCL, IFEval, Nexus FC, …)
  • Baselines compared (from tables):
  • For 16B-class: Qwen3-8B (no think), Ling-mini-2.0, plus preview vs final.
  • For 100B-class: Qwen3-30B-A3B-Instruct-2507, Ling-flash-2.0, plus preview vs final.
  • Inference settings used for LLaDA2.0 evaluations (§6.1):
  • temperature = 0.0
  • block size = 32
  • decoding threshold = 0.95

5.2 Main quantitative results (Tables 1–2; Figure 3; Figure 4; Figure 5)

LLaDA2.0-mini (Table 1)

  • Overall average score: 64.34 vs Ling-mini-2.0 at 65.77, and Qwen3-8B (no think) at 63.42 (Table 1).
  • Selected highlights where LLaDA2.0-mini is strong (Table 1):
  • SQuAD 2.0: 86.50 (vs Ling-mini 75.56)
  • HumanEval: 86.59 (vs Ling-mini 85.98)
  • IFEval-strict-prompt: 80.78 (vs Ling-mini 76.16)
  • GSM8K: 94.24 (near Ling-mini 94.62)
  • MATH: 93.22 (vs Ling-mini 94.66)
  • Noticeable weaker points vs Ling-mini in this table:
  • GPQA: 47.98 (vs 56.80)
  • ZebraLogic: 64.20 (vs 79.85)
  • AIME 2025: 36.67 (vs 47.66)
  • Omni-MATH: 41.70 (vs 48.80)

LLaDA2.0-flash (Table 2)

  • Overall average score: 73.18, close to Qwen3-30B-A3B-Instruct-2507 at 73.60, and above Ling-flash-2.0 at 72.15 (Table 2).
  • Strong coding and agentic results emphasized by the authors are supported by Table 2:
  • HumanEval: 94.51 (vs Qwen 93.29, vs Ling 85.98)
  • MBPP: 88.29 (vs Qwen 86.65, vs Ling 85.01)
  • MultiPL-E: 74.87 (vs Qwen 70.67, vs Ling 65.76)
  • Spider: 82.49 (vs Qwen 81.79, vs Ling 80.58)
  • BFCL v3: 75.43 (vs Qwen 73.19, vs Ling 67.57)
  • AIME 2025: 60.00 (vs Qwen 61.88, vs Ling 55.89)
  • Some weaknesses/oddities visible in Table 2:
  • HARDMath2: 4.27 (ties Qwen 4.27, far below Ling 23.70)
  • BIRD-SQL: 45.76 (below Qwen 47.75 and Ling 47.49)
  • BBH Extra Hard: 27.86 (below Qwen 37.80)

Inference efficiency effects of CAP and diffusion serving (Figure 3; Section 7.3)

  • Tokens per second (TPS) on 4 code/math benchmarks under a consistent setup (§7.3; Figure 3):
  • LLaDA2.0-flash-CAP: 535 TPS
  • LLaDA2.0-flash: 383 TPS
  • Ling-flash-2.0 (AR baseline): 256 TPS
  • Qwen3-30B-A3B-Instruct-2507 (AR baseline): 237 TPS
  • The excerpt summarizes this as “up to 2.1× speed-up over the AR baselines” (§7.3).
  • Tokens per forward (TPF) and average score across 12 benchmarks (Figure 3):
  • The figure caption indicates CAP improves tokens-per-forward while maintaining competitive score; however, the excerpt does not list the per-model numeric score/TPF table beyond the plotted values.

Hyperparameter trade-offs for decoding (Figure 4; Section 6.3)

Using LLaDA2.0-mini on a subset of benchmarks:

  • Denoising threshold (block size fixed at 32):
  • threshold = 0.95: score 70.15, TPF = 2.55 (slowest but best score)
  • threshold = 0.85: score 67.90, TPF = 3.31 (fastest but quality drop) (§6.3; Figure 4)
  • Block size (threshold fixed at 0.95):
  • block size = 16: score 70.26, TPF = 2.44
  • block size = 32: score 70.15, TPF = 2.55
  • block size = 64: described as suboptimal (worse score and speed than size 32) (§6.3; Figure 4)

Long-context evaluation (Figure 5; Section 6.3)

  • On RULER, within native context length up to 32k tokens:
  • LLaDA2.0-flash maintains score >93 from 4k to 32k (§6.3; Figure 5).
  • LLaDA2.0-mini declines from 93.29 at 4k to 83.94 at 32k (§6.3; Figure 5).
  • They extend to 64k using dynamic RoPE scaling via YaRN with scaling factor 2.0, but report a degradation for both models without providing the exact 64k scores (§6.3).

5.3 Do the experiments support the claims?

  • Supportive evidence present in the excerpt
  • Tables 1–2 show that the final LLaDA2.0-mini and LLaDA2.0-flash are competitive with strong AR baselines on a wide benchmark suite, with particularly strong results in coding and BFCL-style tool/agent evaluation (Table 2).

  • The speed comparisons in Figure 3 / §7.3 are directly aligned with the motivation (parallel decoding efficiency), and show substantial throughput gains for the CAP-trained 100B model.

  • What is missing or under-specified (given only this excerpt)

  • There is no detailed ablation table quantifying the individual contributions of WSD vs document-level masking vs top-k merge vs complementary masking vs CAP, beyond qualitative statements (e.g., §4.2 claims document-level mask is “more fundamental” than other tricks, but no numbers are shown here).
  • Training compute, data volume, and hyperparameters are largely absent, making it hard to judge cost/performance trade-offs rigorously.

6. Limitations and Trade-offs

  • Dependence on a strong AR base

  • The whole paradigm assumes access to a high-quality pretrained AR checkpoint (Ling-mini-2.0, Ling-flash-2.0, §4.1; Figure 2). If the base model is weak or misaligned with the target domain, conversion may inherit those weaknesses.

  • Likelihood intractability and approximation

  • Preference alignment replaces true log-likelihood with an ELBO-like reconstruction objective (Eq. (7)–(8), §5.3). This is practical, but it is an approximation, and the excerpt does not analyze when this surrogate might mis-rank preferences.

  • Inference still involves iterative denoising

  • Even with parallel acceptance, block diffusion generation is a multi-step refinement procedure (§5.4). The method trades sequential token-by-token decoding for fewer, heavier forward passes that try to fill many tokens at once; effectiveness depends on confidence calibration (hence CAP).

  • Hyperparameter sensitivity

  • Figure 4 shows clear speed/quality sensitivity to the decoding threshold and block size (§6.3). Deployment likely requires tuning for the target latency/quality envelope.

  • Long-context extension degrades

  • Extending from native 32k to 64k via YaRN scaling factor 2.0 reduces accuracy (§6.3). This suggests limitations in extrapolation beyond the trained context window (even if it remains usable).

  • Reproducibility gaps in the provided excerpt

  • Critical training details are missing here: optimizer, LR schedule, batch sizes, model architecture specifics (layers/hidden/heads), total tokens, compute budget, and hardware. Without these, reproducing the conversion cost and stability properties is difficult.

7. Implications and Future Directions

  • Field-level implication

  • The work provides a concrete recipe for bringing diffusion LMs into the frontier scale regime (100B total parameters, abstract) by leveraging AR training maturity rather than competing with it from scratch. If robust, this shifts diffusion LMs from “small-scale alternative” toward “deployable high-throughput assistants.”

  • What follow-up research it enables (grounded in the excerpt)

  • More systematic ablations: quantify which components most affect stability and quality: WSD schedule shape, document-level masking, checkpoint merging, and CAP weighting λ.
  • Better alignment/RL for diffusion: the Related Work (§2.3) frames RL for diffusion as nascent; the DPO ELBO formulation here could be extended or compared to other diffusion-alignment objectives.

  • Further inference acceleration: §8 suggests pushing decoding speed “to its extreme,” and the inference design (§5.4) plus CAP (§5.2) indicates a path: improve confidence prediction/acceptance so fewer refinement steps are needed.

  • Practical applications / downstream use cases suggested by results

  • Given the strong coding and agent benchmark performance for LLaDA2.0-flash (Table 2: HumanEval 94.51, MBPP 88.29, BFCL v3 75.43), the approach appears particularly suited to:

    • code generation and code reasoning,
    • tool/function calling and agentic workflows,
    • math reasoning benchmarks where it matches strong baselines (e.g., AIME 2025 60.00 close to Qwen’s 61.88, Table 2).

  • Repro/Integration Guidance (based on provided content)

  • Prefer this approach when:
    • You already have a strong AR base model and want to retain its knowledge while improving throughput via parallel decoding (Introduction; §4.1; §7.3).
    • You can benefit from block-wise generation with KV-cache reuse (motivated in §4.1 and §7.3).
  • Deployment knobs explicitly analyzed here:
    • Use block size = 32 as a speed/quality balance (Figure 4; §6.3).
    • Use threshold = 0.95 to maximize quality at some speed cost (Figure 4; §6.3).
    • Consider CAP training if throughput is a priority: it increases TPS substantially in their setup (535 TPS vs 383 TPS without CAP, Figure 3 / §7.3).
  • Integration into serving stacks:
    • They adapt dInfer and add block diffusion support into SGLang to reuse AR-style optimizations like KV-cache reuse (§7.3), implying the inference pattern is close enough to AR decoding to benefit from similar systems techniques.