Skip to content

Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

ArXiv: 2506.08009

🎯 Pitch

Self Forcing introduces a groundbreaking training paradigm for autoregressive video diffusion models, where the model generates each frame by conditioning on its own prior outputs—exactly as it does during inference—rather than relying on ground-truth frames during training. This innovation fundamentally solves the long-standing 'exposure bias' problem, aligning training and inference distributions and enabling real-time, high-quality video generation with sub-second latency. By closing this train-test gap, Self Forcing unlocks efficient and robust streaming video applications and sets a new benchmark for both speed and visual quality in the field.

1. Executive Summary

Self Forcing introduces a training paradigm for autoregressive (AR) video diffusion models that mimics inference during training by rolling out generation on the model’s own past outputs and supervising at the whole-video level. This closes the long-standing train–test mismatch (exposure bias) and enables real-time, low-latency video generation while matching or surpassing the quality of slower, non-causal diffusion models (e.g., 17.0 FPS with 0.69s latency and state-of-the-art VBench scores at 1.3B scale; Table 1).

2. Context and Motivation

  • Problem addressed
  • Exposure bias in sequential generation: during training, AR models are fed ground-truth context; during inference, they must condition on their own imperfect past outputs. This mismatch leads to compounding errors over time (Section 1; Figure 1, middle panels show the mismatch for Teacher Forcing and Diffusion Forcing).
  • In AR video diffusion, prevailing training schemes are:
    • Teacher Forcing (TF): denoise the current frame conditioned on clean, ground-truth past frames (Figure 1a; Figure 2a).
    • Diffusion Forcing (DF): condition on past frames that have been independently noised to various levels (Figure 1b; Figure 2b).

  • Both train on a distribution different from what the model produces at inference (clean past frames produced by the model, not ground truth), which causes error accumulation (Section 1).

  • Why it matters

  • Real-time and interactive use cases—live streaming, gaming, and robotics/world simulation—require low latency and causal processing where the future is unknown (Section 1). Bidirectional diffusion transformers (non-causal) must denoise all frames together and cannot stream; they incur high latency (Section 1).

  • Prior AR alternatives often depend on vector quantization (lossy tokenizers) that hurt fidelity, or add temporal noise at inference to mitigate errors at the cost of temporal consistency and speed (Section 1).

  • Prior approaches and shortcomings

  • Bidirectional diffusion with history guidance or rolling noise schedules can generate long videos but either violate strict causality (future affects past) or pre-generate future content, introducing latency that limits interactivity (Related Work: “Rolling Diffusion and Variants”; Section 2).

  • CausVid combines AR diffusion with Distribution Matching Distillation (DMD), but it computes the video-level loss on DF training outputs that are not from the true inference distribution, so it “matches the wrong distribution” (Section 2; Section 3.3).

  • Positioning of this work

  • Self Forcing trains exactly with the inference recipe: it performs AR rollout with KV caching during training and applies a holistic, whole-video distribution-matching loss on the model’s generated sample (Figure 1c; Figure 2c; Algorithm 1). This directly aligns the training and inference distributions and targets exposure bias head-on (Sections 3.2–3.3).
  • Efficiency is preserved via a few-step diffusion backbone and gradient truncation; additionally, a rolling KV cache enables efficient long-video extrapolation (Sections 3.2 and 3.4).

3. Technical Approach

At a high level, the model is an AR diffusion transformer operating in a latent video space: it generates frame i by denoising noise into an image latent, conditioned on the previously generated frames i−1, i−2, … (Section 3.1). “Autoregressive” means the joint probability of a video is factorized by the chain rule: p(x1:N) = ∏i p(xi | x<i). “Diffusion” means each frame is generated by progressively denoising from Gaussian noise via a few (here 4) steps.

Step-by-step methodology

1) Base model and latent space - The system adopts a transformer-based diffusion backbone with causal attention and text conditioning, operating in a compressed 3D VAE latent space (Section 3.1). The base is Wan2.1-T2V-1.3B with Flow Matching parameterization (Implementation in Section 4; Appendix A).

2) What goes wrong with TF/DF - TF: trains each frame’s denoising conditioned on clean ground-truth past frames (Figure 1a; Figure 2a). - DF: trains each frame’s denoising conditioned on past frames with independent noise levels, hoping to include the inference case in the training distribution (Figure 1b; Figure 2b). - In both, the model never learns to correct its own past mistakes because the context at training time never equals the model’s actual generated context at inference (Section 1; Section 3.3).

3) Self Forcing rollout during training (core mechanism) - During training, the model actually generates a video sample with the exact same AR procedure used at inference: - For frame i, initialize xi at high noise and run the few-step denoising chain while conditioning on the already self-generated “clean” past frames x< i via a KV cache (Figure 2c; Algorithm 1 lines 5–22). - After final denoising for frame i, push its KV embeddings into the cache so that future frames can attend to it efficiently (Algorithm 1 lines 13–14). - This “self-rollout” guarantees that the training sample is drawn from the true inference-time model distribution pθ(x1:N) (Section 3.2).

4) Few-step diffusion + gradient truncation for efficiency - Few-step backbone: 4 denoising steps per frame with a uniform schedule [1000, 750, 500, 250] (Appendix A). - Gradient truncation: - Only the final denoising step for each frame is backpropagated; earlier steps are forward-only to save memory (Algorithm 1 lines 8–12 vs. 16–19; Section 3.2). - Random step supervision: at each training iteration, uniformly sample a denoising step s ∈ {1…T}; treat the output at step s as the final output. This gives supervision “coverage” across all intermediate steps without backprop through the whole chain (Algorithm 1 line 4; Section 3.2). - Detach past frames: gradients do not flow into KV cache entries from prior frames; this stops backpropagation through long temporal histories (Algorithm 1 lines 12–14; Section 3.2).

5) Holistic, video-level distribution matching - Having produced a full video sample via self-rollout, Self Forcing applies a sequence-level distribution-matching objective D(pdata(x1:N) || pθ(x1:N)) rather than per-frame denoising loss (Section 3.3). To stabilize learning, both real and generated videos are diffused to a noise level t before comparison (noise-injection “forward process”); the method then matches pdata,t and pθ,t (Section 3.3). - Three interchangeable objectives are implemented (Section 3.3; Appendix A): - DMD (Distribution Matching Distillation): minimizes reverse KL via score difference between a “real score” network and a “fake score” network; implemented as an MSE on a denoised target (Appendix A, Eq. (2)–(3)). - SiD (Score Identity Distillation): minimizes Fisher divergence between scores (Appendix A, Eq. (4)), typically with α=1 for stability. - GAN: a critic trained with relativistic GAN loss and finite-difference R1+R2 regularization in noise space (Appendix A, Eq. (5)–(7)); they use R3GAN.

  • Why this matters: TF/DF can be seen as minimizing per-frame KL divergences E{x<i}~pdata KL(pdata(xi|x<i) || pθ(xi|x<i)), possibly with noisy contexts sampled from p̃data in DF (Section 3.3 and footnote 1). Self Forcing instead matches the full joint pθ(x1:N) to pdata(x1:N), using context x<i that is sampled from the model itself, which aligns training and inference distributions and tackles exposure bias at its root (Section 3.3).

6) Rolling KV cache for long videos - Challenge: Sliding-window extrapolation typically requires recomputing KV cache or discarding context, both inefficient or harmful to temporal consistency (Section 3.4; Figure 3). - Solution: Maintain a fixed-size KV cache of the last L frames; when adding a new frame, evict the oldest without recomputation (Algorithm 2; Figure 3c). Complexity becomes O(T·L), versus O(T·L^2) without KV or O(L^2+T·L) with recomputation (Figure 3). - Practical fix for artifacts: The first frame’s latent has different statistics than later frames; when the rolling KV cache eventually drops this frame, naïve models flicker. During training, restrict attention so the current chunk cannot attend to the very first chunk when denoising the last chunk, simulating the rolling-cache condition (Section 3.4; Appendix B; Figure 7).

Design choices and rationale - Train-time KV caching with full attention kernels (FlashAttention-3) keeps training fast and simple, avoiding custom causal masks that TF/DF require (FlexAttention) (Section 4 “Training efficiency” and Figure 6). - Few-step diffusion strikes a cost–quality balance; random-step supervision ensures all steps learn without long backprop chains (Section 3.2). - Using sequence-level distribution matching directly optimizes what matters for AR video: the quality of whole generated sequences under the true inference distribution (Section 3.3).

4. Key Insights and Innovations

  • Training-time autoregressive self-rollout with KV caching (fundamental)
  • Novelty: Generation is rolled out during training exactly as at inference (Algorithm 1; Figure 2c), rather than training on parallelized masked batches with ground-truth or noised history (Figure 2a–b).

  • Significance: Eliminates the core distribution mismatch behind exposure bias; the model learns to recover from its own mistakes (Section 3.2–3.3).

  • Holistic video-level distribution matching on the true model distribution (fundamental)

  • Novelty: Compute D(pdata(x1:N) || pθ(x1:N)) on self-generated videos (with noise injection for stability), not frame-wise losses on TF/DF-produced training distributions (Section 3.3).

  • Significance: Aligns the training objective with the actual inference behavior; improves long-horizon stability and reduces error accumulation (Table 2, frame-wise vs chunk-wise robustness).

  • Efficiency via few-step diffusion, gradient truncation, and full-attention kernels (practical but impactful)

  • Novelty: Random-step supervision + truncating gradients to the final step per frame enable sequential training without exploding memory/time; training uses optimized full attention (FlashAttention-3) (Section 3.2; Figure 6).

  • Significance: Sequential post-training attains similar per-iteration times to TF/DF, yet yields better quality for the same wall-clock time (Figure 6 right).

  • Rolling KV cache that avoids recomputation and handles distribution shift (practical)

  • Novelty: A true rolling cache with eviction (Algorithm 2) yields O(T·L) inference for infinite videos; a targeted training tweak prevents flicker when the initial frame falls out of cache (Section 3.4; Figure 3c; Appendix B Figure 7).

  • Significance: Efficient, consistent long video generation suitable for streaming or persistent simulations.

  • Data-free AR conversion path (incremental but useful)

  • DMD/SiD variants can convert a pre-trained bidirectional diffusion model into an AR model “without any video training data,” using a pre-trained score network as the “real” distribution (Section 4, Implementation; Appendix A). This reduces data requirements for post-training.

5. Experimental Analysis

Evaluation setup - Base and training pipeline (Section 4; Appendix A) - Initialize from Wan2.1-T2V-1.3B (flow-matching) and first finetune it for causal attention using 16k ODE solution pairs. - Use 4-step diffusion; implement frame-wise AR and chunk-wise AR (3 latent frames per chunk). - Distribution matching objectives: - DMD and SiD: “data-free” in that they rely on a pre-trained score network (Wan2.1-14B or 1.3B) and the model’s own samples; no real video dataset is needed for these variants (Section 4). - GAN: trains a critic on 70k videos generated by Wan-14B and also uses this data to fine-tune many-step DF/TF baselines; R3GAN objective with R1+R2 regularization (Section 4; Appendix A). - Metrics and hardware (Section 4) - VBench (16 sub-dimensions; Appendix C visualization). - Human preference on 1003 MovieGenBench prompts, one rater per prompt (Section 4; Figure 4; Appendix E). - Throughput (FPS) and first-frame latency on a single NVIDIA H100 GPU, emphasizing both for “real-time.”

Main results - Quality, speed, and latency (Table 1; Figure 4; Figure 5) - Chunk-wise Self Forcing (1.3B, 832×480): - Throughput and latency: 17.0 FPS, 0.69s. - VBench: Total 84.31, Quality 85.07, Semantic 81.28. - Frame-wise Self Forcing: - 8.9 FPS, 0.45s latency. - VBench: Total 84.26, Quality 85.25, Semantic 80.30. - Comparisons (Table 1): - Versus Wan2.1 (same scale, bidirectional diffusion): 0.78 FPS and 103s latency vs 17.0 FPS and 0.69s, with the Self Forcing chunk-wise model slightly higher VBench Total (84.31 vs 84.26). - Versus LTX-Video (efficient diffusion): LTX has 8.98 FPS but 13.5s latency; Self Forcing is ~2× the FPS and ~20× lower latency with higher VBench (84.31 vs 80.00). - Versus SkyReels-V2 and MAGI-1 (other AR-hybrid methods): Self Forcing achieves markedly better throughput and orders-of-magnitude lower latency with higher VBench. - Versus CausVid (Wan-1.3B init): both show 17.0 FPS and 0.69s latency, but Self Forcing achieves higher VBench (84.31 vs 81.20) and better user preference. - Human preference (Figure 4, overall better video): - “Ours” wins: 66.1% vs CausVid, 62.7% vs Wan2.1, 57.9% vs SkyReels-V2, 54.2% vs MAGI-1.

  • Qualitative stability (Figure 5)

  • CausVid exhibits error accumulation (oversaturation over time). Self Forcing remains stable across time steps while matching or slightly exceeding Wan2.1/SkyReels-V2 quality.

  • Ablations (Table 2)

  • Chunk-wise AR:
    • Many-step DF/TF: VBench Total 82.95 (DF), 83.58 (TF).
    • Few-step AR with TF/DF + DMD: 82.32–82.76.
    • Self Forcing: 84.31 (DMD), 84.07 (SiD), 83.88 (GAN).
  • Frame-wise AR (harder; more AR steps, more exposure bias):
    • Many-step DF/TF: 77.24 (DF), 80.34 (TF).
    • Few-step TF/DF + DMD: 78.12–80.56.
    • Self Forcing: 84.26 (DMD), 83.54 (SiD), 83.27 (GAN).

  • Takeaway: Self Forcing consistently outperforms alternatives across objectives and is robust when moving from chunk-wise to frame-wise AR, indicating improved resistance to error accumulation.

  • Rolling KV cache (Section 4 “Rolling KV cache”)

  • Without rolling cache, recomputing KV during sliding windows drops throughput to 4.6 FPS for 10-second videos (inefficient).
  • Naïve rolling cache yields artifacts when the first frame latent leaves the cache.

  • Training with the local-attention restriction resolves artifacts while keeping high throughput: 16.1 FPS (Appendix B; Figure 7).

  • Training efficiency (Figure 6)

  • Per-iteration time comparable among TF, DF, and Self Forcing variants despite Self Forcing being sequential at the frame level (Figure 6 left).
  • For equal wall-clock budgets, Self Forcing reaches higher VBench (Figure 6 right), attributed to full-attention kernels (FlashAttention-3) and avoided masking overhead used by TF/DF (Section 4).

Assessment of evidence - The combination of quantitative metrics (VBench), user study, and speed/latency benchmarks strongly supports the central claims: - Real-time capability at competitive or better quality (Table 1). - Reduced error accumulation visible in qualitative comparisons and in robustness across frame-wise vs chunk-wise settings (Figure 5; Table 2). - Training practicality and efficiency (Figure 6). - Robustness checks include multiple objectives (DMD/SiD/GAN), both AR granularities, and long-video extrapolation with a targeted fix (Appendix B).

6. Limitations and Trade-offs

  • Dependence on a strong base model and teacher components

  • Self Forcing is a post-training stage; it assumes a capable pre-trained diffusion backbone. DMD/SiD moreover leverage a pre-trained “real score” network (Wan2.1-14B or 1.3B) (Appendix A Table 3), which may not always be available or aligned with the deployment domain.

  • Long-horizon limits

  • The method mitigates error accumulation within the training context, but quality still degrades on videos “substantially longer than those seen during training” (Section 5, Limitation).

  • Gradient truncation trade-off

  • Truncating gradients to the final step per frame and detaching across frames reduces memory and enables speed, but may limit learning of very long-range temporal dependencies (Section 5).

  • Objective- and compute-related considerations

  • GAN training requires a generated dataset (70k videos) and careful regularization; DMD/SiD avoid real data but rely on teacher scores and involve training an auxiliary “fake score” head (Appendix A).

  • Few-step diffusion is an approximation; while effective here, extremely high-fidelity demands might benefit from more denoising steps, at the cost of speed.

  • Rolling KV cache assumptions

  • The local-attention training fix is tailored to the statistical mismatch of the first latent; other datasets or VAEs might require different adjustments (Section 3.4; Appendix B).

7. Implications and Future Directions

  • Shift in training paradigm for sequential generative models

  • This work substantiates a “parallel pre-training + sequential post-training” recipe: scale models in the usual parallel fashion, then align train and inference distributions via self-rollout and sequence-level losses (Discussion, Section 5). This directly targets exposure bias and may generalize to other continuous sequence domains (audio, motion, robotics control).

  • Enabling real-time interactive video generation

  • With 17 FPS and sub-second latency at 1.3B parameters (Table 1), the method opens practical use in:

    • Live content creation and streaming where feedback loops matter.
    • Interactive game simulation and world models that must respond frame-by-frame (Section 1).
    • Robotics or action models needing causal, low-latency visual synthesis (Section 2, Related Work [42, 96, 101]).

  • Research directions

  • Even longer contexts and memory: explore architectures with built-in recurrence or state-space models to complement gradient truncation (Section 5; references [19, 63]).
  • Broader objectives and training signals: extend sequence-level matching with hybrid adversarial–score objectives or reinforcement learning for task-conditioned generation.
  • Better extrapolation strategies: beyond the local-attention restriction, devise principled ways to handle non-stationary statistics across very long horizons.

  • Cross-domain generalization: the Self Forcing recipe may benefit AR-diffusion hybrids in language, speech, and multimodal modeling (Related Work, references [1, 53, 90, 110]).

  • Societal considerations

  • Real-time high-quality generation lowers barriers to misuse (deepfakes). The paper explicitly calls for watermarking, detection, and governance to mitigate harms (Appendix D).

Core takeaway: By training exactly the way the model will be used—rolling out its own predictions and aligning the full sequence distribution—Self Forcing turns AR video diffusion into a robust, real-time generator without sacrificing quality. Figures 1–3 and Algorithms 1–2 show how the system matches inference during training; Table 1, Table 2, and Figure 6 demonstrate the resulting accuracy, stability, and efficiency.