Skip to content

BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

ArXiv: 2512.12087

Pitch

BLASST tackles the quadratic bottleneck of LLM attention by introducing a training-free, drop-in sparse mechanism that reuses online softmax statistics to skip negligible computation blocks. The method delivers end-to-end inference acceleration—achieving 1.52× speedup for prefill and 1.48× for decode on modern GPUs—while maintaining accuracy, making it the first practical sparse attention solution that works across all phases without pre-computation overhead.

1. Executive Summary

BLASST (BLocked Attention Sparsity via Softmax Thresholding) introduces a training-free, dynamic sparse attention mechanism that accelerates both prefill and decode phases of LLM inference by reusing online softmax statistics to skip negligible attention blocks. The method requires only a single scalar threshold parameter, achieves 1.52× speedup for prefill at 71.9% sparsity and 1.48× speedup for decode at 73.2% sparsity on Blackwell GPUs, and maintains accuracy within 1% of dense baselines across RULER and LongBench benchmarks—making it the only sparse attention method that simultaneously accelerates both phases without training or pre-computation overhead.

2. Context and Motivation

The Quadratic Attention Bottleneck

The self-attention mechanism in Large Language Models (LLMs) has computational and memory complexity of \(O(n^2)\) for sequence length \(n\). This becomes critical as modern models push toward longer context windows—DeepSeek-R1 and Qwen3 support 128K tokens, with some models reaching 1M tokens. Processing such sequences is computationally prohibitive: attention dominates both latency and memory consumption during inference.

While FlashAttention optimized memory bandwidth through tiling and kernel fusion, it still computes the full attention matrix. The fundamental quadratic complexity remains unaddressed. For deployment scenarios like processing entire codebases, analyzing lengthy documents, or maintaining extended conversations, this bottleneck severely limits practical utility.

Five Barriers to Adopting Existing Sparse Attention Methods

The paper identifies why existing sparse attention approaches have not seen widespread deployment:

  1. Expensive pre-computation: Methods like MInference and XAttention require computing importance scores before determining sparsity patterns, often negating theoretical speedups. MInference uses pre-computed importance metrics, while XAttention ranks anti-diagonal blocks—both requiring passes that add overhead.

  2. Training requirements: Methods like SeerAttention and DeepSeek Sparse Attention (DSA) require model fine-tuning or training entirely new architectures. DuoAttention introduces new layers requiring fine-tuning.

  3. Phase-specific optimization: Most methods target either prefill OR decode exclusively. H2O, SnapKV, Quest, and RocketKV accelerate only decode. MInference, SpargeAttention, and XAttention accelerate only prefill. Neither provides end-to-end acceleration.

  4. Lack of modern hardware support: Many methods lack optimized kernels for current GPUs (Hopper H200, Blackwell B200), making real-world speedup claims unverified.

  5. Intrusive integration: Methods often require substantial modifications to model architectures, attention interfaces, or existing framework APIs.

Where Prior Work Falls Short

Table 1 in the paper provides a comprehensive comparison. Key observations:

Compute-optimized methods (MInference, FlexPrefill, XAttention, SpargeAttention) focus on prefill but use proxy scores or pre-computation that adds overhead. SpargeAttention is most similar to BLASST but differs critically: it uses a separate prediction step rather than reusing already-computed statistics, targets prefill only, and does not skip Value loading for decode.

Memory-optimized methods (H2O, TOVA, StreamingLLM, Quest, RocketKV) focus on KV cache compression and decode acceleration but ignore prefill. These methods reduce memory footprint but often require eviction decisions that may discard important context.

Training-aided methods (SeerAttention, DSA, NSA) achieve high sparsity through learned gates but add training complexity and may show mixed downstream performance.

BLASST's Position

BLASST positions itself as a drop-in replacement for dense attention that:

  • Requires zero training or fine-tuning—works with any pretrained model
  • Requires zero pre-computation—skip decisions happen dynamically during the forward pass
  • Accelerates both prefill and decode phases
  • Provides optimized kernels for Hopper and Blackwell GPUs
  • Integrates with existing APIs—requiring only a single scalar threshold input

The key insight: during FlashAttention's block-wise online-softmax computation, we already compute running maximum statistics. These can identify blocks whose contribution to the final output is negligible without any additional computation.

3. Technical Approach

3.1 Reader Orientation

BLASST is a modification to the FlashAttention kernel that dynamically skips attention blocks during inference based on a simple threshold comparison using already-computed softmax statistics. The system solves the adoption barrier problem of existing sparse attention methods by requiring no pre-computation, no training, and providing a single-threshold interface while still achieving substantial speedups.

3.2 Big-picture Architecture (Diagram in Words)

The BLASST system has three major components:

  1. Modified FlashAttention Kernel — processes attention in blocks as usual, but adds a skip decision after computing each block's QK^T scores based on comparing the block's local maximum to a running maximum.

  2. Calibration Procedure — determines the optimal threshold λ for a target sparsity level by fitting an exponential model λ·L = α·exp(β·s) from a single forward pass over a calibration dataset.

  3. Hardware-Specific Kernel Implementations — two specialized implementations: a prefill kernel that skips softmax computation and MMA operations (targeting compute-bound scenarios), and a decode kernel that skips Value loads from HBM (targeting memory-bound scenarios).

Information flow: Input Q, K, V blocks → Process each block sequentially → Compute QK^T scores and local maximum → Compare to running maximum minus threshold → Skip or execute softmax + Value multiplication → Accumulate output → Final normalization.

3.3 Roadmap for the Deep Dive

  • First, the mathematical foundation of the pruning criterion—how the online softmax running maximum enables skip decisions without pre-computation.
  • Second, the algorithm design—exactly what operations are skipped and how the decision is made per-block.
  • Third, the calibration procedure—how thresholds relate to context length and sparsity, and how to select them automatically.
  • Fourth, the kernel design—why prefill and decode require different optimization strategies and how the pipeline schedules change.
  • Fifth, sparsity-aware training—how models can be trained to be more robust to sparse attention as an optional extension.

3.4 Detailed, Sentence-based Technical Breakdown

This is a systems/efficiency paper that modifies the FlashAttention kernel to dynamically prune attention computation with zero overhead.

The Core Mathematical Insight: Online Softmax Enables Zero-Cost Pruning

To understand BLASST, we must first understand FlashAttention's online softmax computation. In standard attention, we compute:

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V\]

The softmax operation normalizes attention scores so they sum to 1. FlashAttention processes this in blocks (tiles) to minimize memory transfers. For each block, it maintains:

  • \(m_i^{(j)}\): the running maximum of attention scores seen so far for row \(i\) after processing \(j\) blocks
  • \(\tilde{m}_i^{(j)}\): the local maximum within the current block \(j\)

The key insight is that softmax is dominated by its maximum value. After the running maximum \(m_i^{(j)}\) is established, if a subsequent block's local maximum \(\tilde{m}_i^{(j)}\) is much smaller, the block's contribution after softmax will be near zero.

The pruning criterion: If \(\tilde{m}_i^{(j)} - m_i^{(j)} < \ln(\lambda)\) for some threshold \(\lambda\), then:

\[\exp\left(\tilde{m}_i^{(j)} - m_i^{(j)}\right) < \lambda \approx 0\]

Since the maximum value in the block is bounded by \(\lambda\) relative to the running maximum, all post-softmax values in this block will be negligible.

Why this is zero-overhead: FlashAttention already computes \(m_i^{(j)}\) and \(\tilde{m}_i^{(j)}\) for numerical stability. BLASST simply adds one comparison per block to decide whether to skip—no additional computation beyond what FlashAttention already performs.

Algorithm Design: What Gets Skipped

Algorithm 1 presents the modified FlashAttention forward pass. For each query block \(Q_i\) and key-value blocks \(K_j\), \(V_j\):

Step 1: Compute attention scores. Calculate \(S_{ij} = Q_i K_j^\top\) (line 4). This is the QK matrix multiplication, typically using tensor cores.

Step 2: Compute local and running maximums. The local maximum \(\tilde{m}_i^{(j)} = \text{rowmax}(S_{ij})\) (line 5). Update the running maximum: \(m_i^{(j)} = \max(m_i^{(j-1)}, \tilde{m}_i^{(j)})\) (line 6).

Step 3: Skip decision. If \(\tilde{m}_i^{(j)} - m_i^{(j)} < \ln(\lambda)\), continue to the next block (lines 7-9). This single comparison determines whether the block is negligible.

Step 4: Compute softmax and accumulate. If not skipped, compute \(\tilde{P}_{ij} = \exp(S_{ij} - m_i^{(j)})\) (line 10), update the running sum \(l_i^{(j)}\) (line 11), and accumulate the output \(O_i^{(j)} = \tilde{P}_{ij} V_j\) (line 12).

What is saved when skipping:

  1. CUDA core operations: The exponential function exp(·) requires multiple instructions: MUFU.EX2 (exponential), FMUL (multiplication), and FADD (addition). For a typical block, this saves thousands of CUDA core instructions.

  2. Tensor core operations: The matrix multiplication \(\tilde{P}_{ij} V_j\) (attention weights × values). In compute-bound prefill, avoiding MMA operations provides substantial speedup.

  3. Memory bandwidth: Loading the Value block \(V_j\) from HBM to SRAM. This is critical for memory-bound decode.

Design choice: Block-level vs. token-level pruning. The ideal would compare each token's score \(S_{ij}\) against the global maximum, but this is too expensive. The block-local maximum approximation enables efficient kernel-level decisions while still capturing the important information—since attention patterns are often sparse, blocks containing important tokens will have high local maxima.

The Three-Step Approximation Intuition

The paper provides an intuitive framing of the approximation chain:

  1. Ideal importance: Each score's importance is its value relative to the global maximum (unknown).

  2. Running maximum proxy: The running maximum \(m_i^{(j)}\) serves as a tractable approximation to the global maximum, computed incrementally.

  3. Block-level decision: Replace token-level \(S_{ij}\) with block-local maximum \(\tilde{m}_i^{(j)}\) for efficient per-block decisions.

This approximation works because attention scores are highly non-uniform—a few tokens receive most of the attention weight. Blocks containing high-scoring tokens will have high local maxima; blocks far from important information will have uniformly low scores.

Calibration: The Threshold-Context Relationship

A critical deployment challenge: what threshold \(\lambda\) should be used? The paper reveals that threshold must be calibrated for context length.

The problem: Figure 2 (right) shows that for a fixed threshold, sparsity varies wildly with context length. To achieve 75% sparsity, you need \(\lambda \approx 10^{-4}\) for 8K contexts but \(\lambda \approx 10^{-5}\) for 64K contexts.

The relationship: Through empirical analysis, the paper finds an inverse proportional relationship:

\[\lambda = \frac{a}{L}\]

where \(L\) is context length and \(a\) is a model-specific scale factor.

Theoretical grounding: Attention scores are row-normalized to sum to 1. Longer sequences have lower average scores per token, requiring proportionally smaller thresholds to achieve the same relative pruning.

The calibration procedure (Algorithm 2):

  1. Collect calibration samples \(D = \{(x_i, L_i)\}_{i=1}^N\) across different context lengths.
  2. For each sample and candidate threshold, measure achieved sparsity.
  3. Fit the exponential model \(\lambda \cdot L = \alpha \cdot \exp(\beta \cdot s)\) where \(s\) is sparsity.

The exponential form reflects heavy-tailed attention distribution: small threshold increases prune many low-scoring blocks, but further increases yield diminishing returns.

Key result: Table 6 shows calibration reduces sparsity variance from ±25% (fixed threshold) to ±1.2% (calibrated \(\lambda = a/L\)) across context lengths.

Kernel Design: Why Prefill and Decode Differ

Prefill and decode have fundamentally different performance characteristics, requiring specialized kernel optimizations.

Prefill (compute-bound):

  • The kernel is bottlenecked by CUDA cores (softmax) and tensor cores (matrix multiplication), NOT memory bandwidth.
  • Solution: Skip both softmax computation and MMA operations for pruned blocks.
  • Value blocks remain loaded from HBM because: (1) memory bandwidth isn't the bottleneck, (2) prefetching benefits from predictable access patterns, and (3) conditional loading would add latency.

Decode (memory-bound):

  • The kernel is bottlenecked by HBM bandwidth to fetch the KV cache, since attention involves only a single Query against all Keys.
  • Solution: Skip the memory-intensive load of Value blocks \(V_j\) for pruned blocks, directly cutting memory traffic.
  • Challenge: In naive implementation, Value loads happen before skip decisions are made.
  • Solution: Batched load scheduling—process multiple consecutive QK^T products back-to-back (\(K_1^\top Q, K_2^\top Q, \ldots, K_B^\top Q\)), then issue batched loads only for Value tiles that pass the threshold check.

Pipeline Schedule Visualization

Figure 3 and Figure 4 show pipeline schedules as timeline diagrams.

Prefill pipeline (Figure 3):

  • Normal FlashAttention (3a): 18 time units for 4 loop iterations.
  • BLASST (3b): 14 time units, skipping softmax and MMA for loops 1 and 3.
  • MMA warp handles BMM1 (QK^T) and BMM2 (attention×values).
  • Softmax warpgroups handle exponentiation (EX2), row sums, and scaling.
  • Skipping frees execution units, allowing subsequent operations to schedule earlier.

Decode pipeline (Figure 4):

  • Normal FlashAttention (4a): 38 time units for Value loads.
  • BLASST (4b): 31 time units, skipping Value loads for loops 1, 2, and 4.
  • Arrows show scoreboard dependencies from skip check after BMM1.
  • Batched K loads allow conditional V loads without pipeline bubbles.

Skip Decision Implementation Details

The decision process requires only a few instructions per block:

  1. Predicate setting: Each thread sets a predicate based on the threshold comparison.
  2. Warp vote: Issue a VOTE instruction to determine if all threads in a warp agree to skip.
  3. Block-level coordination: One thread per warp issues an ATOMIC instruction to shared memory to coordinate across the softmax warpgroup.

These instructions are hidden behind existing operations, adding negligible latency overhead. The paper verifies this with 0% sparsity measurements (Table 5): kernels achieve 0.96-1.00× baseline, confirming the skip check is fully overlapped with tensor core or HBM load instructions.

Sparsity-Aware Training (Optional Extension)

While BLASST is training-free by design, the paper explores sparsity-aware training to push the accuracy-sparsity frontier.

Method: During fine-tuning, apply BLASST in the forward pass. In the backward pass, skipped blocks receive no gradients (since they weren't computed). This encourages the model to concentrate important information in high-scoring blocks.

Result (Figure 6): At 50-75% sparsity, sparse-trained models reduce accuracy degradation by up to 1.7× compared to training-free application.

Extensibility to Attention Variants

BLASST depends only on tiled online softmax, making it compatible with:

  • MHA (Multi-Head Attention): Standard attention, fully supported.
  • MQA (Multi-Query Attention): Multiple query heads share one KV pair, supported.
  • GQA (Grouped Query Attention): Groups of query heads share KV pairs, supported.
  • MLA (Multi-Head Latent Attention): Projects KV to latent space, still uses online softmax within latent space, supported (Table 11 shows DeepSeek-R1 results).

For MLA, which shifts decode toward compute-bound, BLASST provides benefits regardless of bottleneck type by eliminating both computation and memory accesses.

Summary of Design Choices

Choice Rationale
Running maximum vs. global maximum Zero overhead—already computed for numerical stability
Block-level vs. token-level decisions Efficient kernel implementation with single comparison per block
Skip softmax + MMA (prefill) Targets compute-bound bottleneck
Skip Value loads (decode) Targets memory-bound bottleneck
Batched load scheduling (decode) Avoids pipeline bubbles from conditional loads
Exponential calibration model Captures heavy-tailed attention distribution
Training-free primary design Maximizes adoption potential
Optional sparsity-aware training Pushes accuracy-sparsity frontier when training is acceptable

4. Key Insights and Innovations

Innovation 1: Zero-Overhead Skip Decisions Via Statistics Reuse

The most fundamental innovation is recognizing that FlashAttention's online softmax already computes exactly the statistics needed for pruning decisions—the running maximum \(m_i^{(j)}\) and local maximum \(\tilde{m}_i^{(j)}\). Prior methods like SpargeAttention use separate prediction steps or proxy scores that add computation. BLASST adds only a single comparison per block, which the kernels hide behind tensor core or memory operations, achieving verified zero overhead at 0% sparsity.

This is significant because it eliminates the pre-computation barrier that plagued prior methods. Theoretical speedups from sparse attention often evaporate when the cost of determining sparsity patterns is included. BLASST's speedups are realized end-to-end.

Innovation 2: Unified Algorithm for Both Prefill and Decode

Prior sparse attention methods optimized either prefill OR decode, requiring different techniques for different phases. BLASST provides a single algorithmic framework that works for both, with specialized kernel implementations that target each phase's specific bottleneck:

  • Prefill kernels target compute savings (skip softmax + MMA)
  • Decode kernels target memory savings (skip Value loads)

This matters for production deployment where both phases occur. The same threshold interface and calibration procedure work across both, simplifying integration. Table 1 shows BLASST is the only method with all four properties: accelerates prefill, accelerates decode, no training, no pre-computation.

Innovation 3: The Inverse Threshold-Context Relationship

The calibration analysis reveals a simple but non-obvious relationship: \(\lambda = a/L\). This is theoretically grounded (attention scores sum to 1, so longer sequences have proportionally lower per-token scores) but has important practical implications:

  1. Predictable deployment: A single calibration per model suffices for all context lengths.
  2. No manual tuning: The relationship holds across datasets (Table 12 shows calibration transfers without task-specific retuning).
  3. Interpretable control: Users can target specific sparsity levels and predict compute savings.

Without this insight, fixed thresholds would produce wildly varying sparsity across context lengths, making production deployment unreliable.

Innovation 4: Hardware-Aware Kernel Design With Verified Speedups

The paper provides optimized kernels for both Hopper (H200) and Blackwell (B200) GPUs, demonstrating that theoretical speedups translate to real hardware. The kernel design addresses specific architectural characteristics:

  • Pipeline schedule modifications (Figures 3-4) show exactly how skipping compresses the instruction timeline
  • Batched load scheduling solves the decode challenge of conditional Value loads
  • Verifiable overhead claims: 0% sparsity measurements (Table 5) prove skip checks are hidden

This is more thorough than many prior works that report theoretical FLOP reductions without kernel implementations, or only report results on older hardware.

Innovation 5: Occasional Accuracy Improvements From Pruning

A counterintuitive finding: BLASST sometimes improves accuracy over dense attention (Table 2). For example, Qwen3-8B at 50% sparsity shows MATH500 accuracy of 96.23 vs. 95.87 (dense), and AIME 2024 of 76.50 vs. 75.00.

The paper attributes this to: 1. Implicit denoising: Pruning low-attention blocks concentrates probability mass on relevant tokens 2. Filtering distractions: Long reasoning chains may contain redundant or detrimental intermediate steps

This is significant because it suggests sparse attention isn't just an efficiency hack—it can sometimes be beneficial for quality. This aligns with findings from the wider literature on attention sparsity being an inherent property of language models, not just an approximation.

5. Experimental Analysis

Evaluation Methodology

Models evaluated: - Llama-3.1-8B-Instruct (128K context support) - Qwen3-8B-Instruct (128K context support) - Llama-3.1-70B-Instruct (scalability verification) - Qwen3-30B-A3B-Instruct (scalability verification) - DeepSeek-R1 (MLA compatibility verification)

Datasets: - RULER (Hsieh et al., 2024): Synthetic retrieval and reasoning benchmarks, context lengths 4K-128K, includes NIAH (needle-in-a-haystack), VT (variable tracking), FWE (frequency weighted extraction) - LongBench v2: Real-world QA, summarization, and code completion - Reasoning benchmarks: MATH500, AIME 2024, GPQA, LiveCodeBench - RepoQA: Code repository understanding at extreme lengths (16K, 200K)

Baselines: - Prefill: MInference, FlexPrefill, XAttention - Decode: Quest, RocketKV - Dense: FlashAttention-3 BF16

Hardware: NVIDIA Hopper H200 and Blackwell B200 GPUs

Calibration setup: ~1000 sequences from RULER across context lengths 4K, 8K, 16K, 32K, 64K to fit calibration parameters α and β

Metrics: - Task accuracy (exact match for retrieval, graded responses for reasoning) - Sparsity percentage (fraction of blocks skipped) - Kernel speedup over FlashAttention baseline

Main Quantitative Results

Overall performance (Table 2):

At 75% target sparsity on Llama-3.1-8B: - RULER-32K: 91.67 vs. 92.33 dense (-0.66) - LongBench: 31.80 vs. 31.40 dense (+0.40) - MATH500: 73.89 vs. 73.40 dense (+0.49) - AIME 2024: 46.01 vs. 46.66 dense (-0.65)

At 75% sparsity on Qwen3-8B: - RULER-32K: 92.11 vs. 91.90 dense (+0.21) - LongBench: 34.40 vs. 33.60 dense (+0.80) - MATH500: 96.07 vs. 95.87 dense (+0.20)

Prefill phase comparison (Table 3) on Llama-3.1-8B at 50% sparsity:

Method RULER Average LongBench Overall
Dense 93.21 31.4
FlexPrefill 87.72 25.7
MInference 84.15 31.2
XAttention 92.44 30.6
BLASST 92.87 31.8

BLASST achieves best accuracy among sparse methods, within 0.34 points of dense on RULER and matching dense on LongBench.

Decode phase comparison (Table 4) on Qwen3-8B:

Method RULER-32K MATH500 AIME 2024 Average
Dense 91.90 95.87 75.00 68.57
Quest 56.23 94.18 71.50 60.75
RocketKV 87.89 95.88 73.54 66.91
BLASST 91.55 96.23 76.50 68.97

BLASST matches or exceeds dense baseline while Quest and RocketKV show substantial accuracy degradation on long-context tasks.

Kernel performance (Table 5) on Blackwell B200:

Prefill (64K sequence, batch=1): | Sparsity | Speedup | |----------|---------| | 0% | 1.00× | | 49.2% | 1.33× | | 71.9% | 1.52× | | 88.9% | 1.71× |

Decode (32K sequence, batch=148): | Sparsity | Speedup | |----------|---------| | 0% | 0.98× | | 46.7% | 1.25× | | 73.2% | 1.48× | | 87.0% | 1.71× |

On Hopper H200, prefill achieves 1.52× at 71.0% sparsity; decode achieves 1.40× at 70.5% sparsity.

Calibration results (Table 6):

Target 50% sparsity, Llama-3.1-8B: | Context | Fixed λ | Calibrated λ | |---------|---------|--------------| | 4K | 23.09% (-26.91) | 54.20% (+4.20) | | 64K | 74.63% (+24.63) | 48.75% (-1.25) |

Calibration reduces variance from ±25% to ±1.2%, enabling predictable deployment.

Large model results (Tables 9-10):

Qwen3-30B on LongBench V2 at 70% sparsity: 39.53 vs. 36.28 dense (+3.25 improvement)

Llama-3.1-70B on RULER-hard at 80% sparsity: 97.07% vs. 97.40% dense (-0.33), demonstrating extreme sparsity is feasible on larger models.

Very long sequences (Table 8):

Qwen3-Coder-30B at 200K context: - Prefill sparsity: 57.5% - Accuracy: 0.841 vs. 0.850 dense (-0.9%) - Prefill+Decode combined: 57.5%/40.8% sparsity, 0.838 accuracy

Sparsity-aware training (Figure 6):

At 60% observed sparsity on RULER: - Dense training + sparse inference: ~92.6% - Sparse training + sparse inference: ~93.3% - Improvement: ~1.7× reduction in accuracy degradation

Ablation Studies

Sparsity distribution analysis (Figure 7): Layer 0 shows high variance across heads (some ~20% sparsity, others ~80%). BLASST automatically adapts to this heterogeneity without explicit per-layer or per-head tuning.

Combination with other methods (Table 7): - XAttention (prefill) + BLASST (decode): RULER-16K 92.89 (-0.33 from dense) - BLASST (prefill) + RocketKV (decode): RULER-16K 92.60 (-0.62 from dense)

Composability demonstrated with minimal incremental accuracy loss.

Extreme sparsity (Figure 8): At 80% prefill sparsity on RULER-16K for Qwen3-8B, BLASST maintains ~91.5% accuracy vs. XAttention's ~90.5%. BLASST shows more graceful degradation.

Tile row reordering (Figure 9): Processing tiles in reverse order has negligible impact on VT and FWE tasks, confirming robustness to processing order.

Assessment: Do the Experiments Support the Claims?

Claim 1: BLASST accelerates both prefill and decode without training or pre-computation. Strongly supported. Table 1's feature comparison is accurate. Tables 5 and kernel benchmarks demonstrate real speedups on modern hardware. The 0% sparsity measurements (0.96-1.00×) confirm zero overhead.

Claim 2: Minimal accuracy degradation at target sparsity. Supported across multiple models and datasets. Tables 2-4 show consistent results. Occasional improvements noted but not claimed as guaranteed.

Claim 3: Calibration enables predictable sparsity control. Strongly supported. Table 6 shows calibration reduces variance from ±25% to ±1.2%. Table 12 shows calibration transfers across tasks.

Claim 4: Compatible with attention variants including MLA. Supported. Table 11 shows DeepSeek-R1 (MLA architecture) results with minimal degradation at 60% sparsity.

Claim 5: Scalable to large models and long contexts. Supported. Tables 9-10 show 70B model results. Table 8 shows 200K context results.

Limitations noted:

  1. Single-sentence accuracy losses: Some configs show small accuracy drops (e.g., Llama AIME 2024 at 75%: 46.01 vs. 46.66). These are minor but non-zero.

  2. Calibration requires forward pass: While "single forward pass" is claimed, this still requires inference compute. Not a major issue but worth noting.

  3. Hopper decode speedup lower: Table 5 shows H200 decode at 70.5% sparsity achieves only 1.40× vs. B200's 1.48×. Hardware-specific differences exist.

  4. Sparsity-aware training not fully evaluated: Figure 6 shows results but training methodology details are brief compared to inference contributions.

6. Limitations and Trade-offs

Assumption: The Running Maximum Is a Reliable Proxy for Global Maximum

BLASST's pruning criterion fundamentally assumes that the running maximum \(m_i^{(j)}\) computed incrementally during block processing is a sufficient approximation of the true global maximum for skip decisions. This assumption has a subtle consequence: early blocks set the pruning threshold for later blocks. If important tokens happen to appear late in the sequence (after the running maximum has been established at a lower value), their blocks may be incorrectly skipped.

The paper acknowledges this in Section 3.1.1, describing it as a "three-step approximation." The block-level decision (replacing token-level \(S_{ij}\) with block-local maximum \(\tilde{m}_i^{(j)}\)) is necessary for efficiency but means that a block containing a mix of important and unimportant tokens will be kept based on its local maximum, potentially reducing effective sparsity.

The tile row reordering experiments (Appendix A.4, Figure 9) test whether processing order matters by reversing the sequence—processing tiles from the end first. The results show "dataset-dependent behavior" with "negligible impact" on VT and FWE tasks. This is encouraging for robustness but doesn't fully characterize worst-case scenarios where critical information is distributed non-uniformly across the sequence.

Block-Level Granularity Limits Fine-Grained Sparsity

BLASST operates at the block level (typically 128×128 tiles in FlashAttention), meaning all tokens within a block are either kept or skipped together. This is a coarse granularity compared to token-level pruning methods.

The paper argues this is necessary for kernel efficiency—a single comparison per block enables the zero-overhead design. However, this creates a tradeoff:

  • On one hand: Block-level decisions are efficient and often sufficient because attention tends to be locally concentrated—important tokens often cluster together.
  • On the other hand: A block containing even one high-scoring token will be kept entirely, limiting achievable sparsity on distributions with scattered important tokens.

Figure 7 visualizes sparsity distribution across layers and heads, showing substantial heterogeneity (some heads at ~20% sparsity, others at ~80%). BLASST naturally adapts to this heterogeneity, but the block-level granularity means that even highly sparse heads may have some dense blocks that could theoretically be further pruned with finer granularity.

Calibration Requires Representative Data

The calibration procedure (Algorithm 2) requires a dataset \(D\) of samples across different context lengths to fit the relationship \(\lambda \cdot L = \alpha \cdot \exp(\beta \cdot s)\). The paper uses ~1000 sequences from RULER for calibration.

Assumption: The calibration dataset is representative of the deployment distribution. If the target workload has very different attention patterns (e.g., different domains, languages, or task types), the fitted parameters may not generalize perfectly.

Table 12 shows calibration transfers across RULER subsets (NIAH variants, CWE, QA), suggesting robustness. However, this is still within the same benchmark family. The paper does not evaluate cross-domain transfer—for example, calibrating on general text and deploying on code, or calibrating on English and deploying on multilingual content.

Additionally, calibration requires one forward pass over the calibration dataset. For a 128K-context model on 1000 sequences, this is non-trivial compute. The paper frames this as acceptable because it's a one-time cost per model, but it's worth noting that calibration overhead scales with model size and context length.

Decode Kernel Does Not Skip Key Computation

In the decode kernel design (Section 4.2), BLASST skips Value block loads for pruned blocks but still computes the query-key product (\(K_j^\top Q\)) for all blocks before the skip decision can be made. This is necessary because the skip decision depends on the attention scores.

The implication: Key matrix computation is never skipped. For decode with very large KV caches, the \(K_j^\top Q\) computation may still be substantial. The paper's speedups (1.48× at 73.2% sparsity on decode) reflect actual realized gains including this constraint, but the theoretical maximum speedup is bounded below 100% of the dense computation.

This is a fundamental limitation of the approach—skip decisions require computing at least some of the attention scores. Methods that use separate predictors (like SpargeAttention's prediction step) could potentially skip Key computation too, but at the cost of pre-computation overhead that BLASST explicitly avoids.

Prefill Kernel Does Not Skip Value Loads

Conversely, the prefill kernel design (Section 4.1) skips softmax and MMA operations but does not skip Value block loads from HBM. The paper explains this is because:

  1. Memory bandwidth is not the bottleneck in compute-bound prefill
  2. Prefetching benefits from predictable memory access patterns
  3. Conditional loading latency would exceed savings

This is a reasonable design choice for current hardware, but it means the prefill kernel's speedup potential is bounded by compute savings, not memory savings. If future workloads or hardware architectures shift prefill toward memory-bandwidth-bound regimes (e.g., very large batch sizes, unusual memory hierarchy), the current design may be suboptimal.

Accuracy Degradation at Extreme Sparsity

While BLASST shows impressive accuracy preservation at 50-75% sparsity, degradation becomes more pronounced at higher levels:

  • Table 9 (Qwen3-30B LongBench): At 90% target sparsity, accuracy drops from 47.77 to 45.97 (-1.8 points)
  • Figure 8 shows the accuracy-sparsity curve: at ~80% sparsity, accuracy is ~91.5% (vs. 93.22% baseline)

The paper correctly positions 50-75% as the practical operating range. Users seeking extreme sparsity (>80%) should expect non-trivial accuracy tradeoffs. This is not a failure of the method but a realistic characterization of the accuracy-sparsity frontier.

Notably, sparsity-aware training (Figure 6) improves this frontier but does not eliminate the tradeoff—sparse-trained models still show degradation at high sparsity, just less severe.

No Support for Training From Scratch

BLASST is designed as a training-free inference optimization for existing pretrained models. The sparsity-aware training extension applies only during fine-tuning, not during pretraining from scratch.

This is a deliberate scope limitation (the paper emphasizes "drop-in" deployment), but it means BLASST cannot fundamentally alter attention patterns learned during pretraining. Models pretrained with dense attention may have attention distributions that are less amenable to sparsity than they could be if trained with sparsity from the start.

Methods like DeepSeek Sparse Attention (DSA) and Native Sparse Attention (NSA) that train new architectures from scratch may achieve higher sparsity levels by learning inherently sparse patterns during pretraining. BLASST's training-free approach trades some theoretical optimality for practical deployability.

Limited Evaluation on Non-English Content

All benchmark datasets (RULER, LongBench, MATH500, AIME, GPQA, LiveCodeBench) are English-language or code. The paper does not evaluate multilingual performance or languages with different morphological properties that may affect attention patterns.

Attention sparsity patterns could differ across languages—agglutinative languages, character-based scripts, or languages with different word order might exhibit different attention distributions. The calibration procedure's robustness across domains is unverified for non-English content.

Potential Interaction With Quantization Not Addressed

The paper evaluates BF16 precision but does not address quantization formats increasingly used in deployment (INT8, INT4, FP8, or the NVFP4 format mentioned for DeepSeek-R1 evaluation). Quantization changes the numerical precision of attention scores, which could theoretically affect the threshold calibration.

For example, if attention score distributions shift under quantization (due to rounding effects), the relationship \(\lambda = a/L\) may need recalibration. The DeepSeek-R1 results (Table 11) use NVFP4 but do not analyze whether quantization affects the calibration relationship.

Sparsity-Aware Training Evaluation Is Limited

Section 3.4 and Figure 6 present sparsity-aware training results showing improved accuracy-sparsity tradeoffs, but the methodology is described briefly:

"during fine-tuning, we apply BLASST in the forward pass to skip negligible attention blocks based on the threshold criterion"

Key details not addressed: - What pretraining checkpoint is fine-tuned from? - What fine-tuning dataset is used? - How long is fine-tuning (steps, epochs)? - How is the threshold selected during training?

The results are encouraging but the methodology is less thoroughly characterized than the inference contributions. Users considering sparsity-aware training would need to experiment with hyperparameters not specified in the paper.

7. Implications and Future Directions

How This Work Changes the Landscape

BLASST establishes that zero-overhead dynamic sparse attention is practically achievable. Prior to this work, sparse attention methods faced a fundamental tension: methods that made accurate pruning decisions required pre-computation or training, while training-free methods made cruder decisions. BLASST resolves this by recognizing that FlashAttention's online softmax statistics are exactly what's needed for accurate pruning—no additional computation required.

This shifts the baseline for what constitutes a practical sparse attention method. Future work that proposes pre-computation or proxy scores must now justify why those additional costs are necessary when comparable accuracy can be achieved with zero overhead.

The unified prefill+decode approach changes deployment calculus. Previously, practitioners wanting end-to-end sparse inference had to combine multiple methods (e.g., MInference for prefill + RocketKV for decode) with different APIs, calibration procedures, and compatibility concerns. BLASST provides a single interface for both phases. Table 7 demonstrates composability with other methods, but BLASST alone is sufficient for many use cases.

Calibration simplicity enables production adoption. The inverse relationship \(\lambda = a/L\) means that once a model is calibrated (finding parameter \(a\)), sparsity can be controlled predictably across any context length. This is precisely the kind of reliability that production systems require—operators can specify a target sparsity (e.g., "70%") and trust that it will be achieved regardless of the input sequence length.

Occasional accuracy improvements challenge the "approximation" framing. The observation that BLASST sometimes improves over dense attention (Table 2: Qwen3-8B MATH500 96.23 vs. 95.87) suggests that dense attention may be computing unnecessary or even harmful interactions. This aligns with a growing understanding that LLMs have inherently sparse attention patterns—BLASST exploits this property rather than fighting against it.

Follow-Up Research This Work Enables

1. Adaptive thresholds per-layer or per-head. Figure 7 shows substantial heterogeneity in sparsity across layers and heads—some heads naturally achieve 80% sparsity while others only 20%. Currently, BLASST uses a single global threshold. An extension could calibrate separate thresholds per-layer or per-head, potentially achieving higher overall sparsity while preserving accuracy on the heads that need dense attention.

The paper notes that BLASST "automatically adapts" to this heterogeneity, but a more explicit mechanism could push the Pareto frontier further. The tradeoff is calibration complexity: instead of fitting two parameters \((\alpha, \beta)\), you'd need 2×(number of layers) or 2×(number of heads) parameters.

2. Integration with KV cache compression. BLASST and memory-optimized methods like RocketKV are largely orthogonal (Table 7 shows successful combination). A deeper integration could exploit BLASST's skip decisions to inform KV cache eviction—for example, if a block is consistently skipped across multiple decode steps, its KV entries could be evicted from the cache.

This would create a unified framework where compute sparsity (BLASST) and memory sparsity (KV compression) reinforce each other, potentially achieving multiplicative efficiency gains.

3. Extending the inverse relationship to other model families. The paper demonstrates calibration on Llama-3.1 and Qwen3. A systematic study across diverse architectures (Mistral, Gemma, Phi, DeepSeek, etc.) would test whether the \(\lambda = a/L\) relationship is universal or architecture-specific.

If the relationship holds generally, calibration could become a standard step in model deployment pipelines. If it varies, understanding which architectural factors affect the calibration would inform model design for sparse attention compatibility.

4. Pretraining with BLASST. The sparsity-aware training experiments apply only during fine-tuning. An open question: what if models were pretrained from scratch with BLASST active? Would they learn fundamentally different attention patterns that are even more amenable to sparsity?

This is a more ambitious direction because it requires modifying the pretraining pipeline rather than just fine-tuning. But the potential payoff is models that are "designed for sparsity" from the ground up, potentially achieving higher sparsity at equivalent accuracy.

5. Hardware co-design for conditional execution. The kernel design carefully works around hardware constraints—for example, the decode kernel uses batched load scheduling to avoid pipeline bubbles. Future GPU architectures could provide native support for conditional execution patterns, potentially making the skip decision even cheaper or enabling skipped Key computation.

The pipeline schedule visualizations (Figures 3-4) provide a template for how hardware architects might think about supporting sparse attention: reducing dependencies, enabling conditional memory requests, and providing efficient synchronization primitives.

6. Extending to other attention mechanisms. BLASST is demonstrated on standard attention and MLA. Could the same principle apply to other attention variants?

  • Sliding window attention: The block structure is already well-defined; BLASST could skip blocks outside the window more aggressively.
  • Linear attention: The softmax normalization is replaced by different operations; the running maximum insight may not apply directly.
  • FlashAttention with PagedAttention: The KV cache is non-contiguous; skip decisions may need modification to handle indirection.

Understanding which attention mechanisms are compatible with the BLASST principle would clarify its scope.

Practical Applications and Downstream Use Cases

Long-context serving systems. For services deploying long-context models (e.g., RAG systems, document analysis tools, code completion for large repositories), BLASST provides immediate efficiency gains with minimal integration cost. A single parameter (target sparsity) controls the accuracy-performance tradeoff, making it easy to configure for different latency SLOs.

The calibration procedure is designed for production deployment: run once per model, store the parameters \((\alpha, \beta)\), and the system handles any context length automatically.

On-device inference. Edge devices have stricter memory and compute constraints. BLASST's ability to skip both computation and memory accesses is valuable for running larger models on constrained hardware. The 1.5× speedup at ~70% sparsity could translate directly to longer battery life or support for larger models on the same device.

The training-free nature is particularly valuable here: device manufacturers cannot retrain models but can apply BLASST to any pretrained checkpoint.

Batch inference workloads. For high-throughput batch processing (e.g., evaluating large datasets, generating training data, processing backlogs), BLASST provides predictable speedups. Figure 5 shows TTFT and TPOT speedups in a batched inference setup (in-flight batching with concurrency 64).

Cost reduction for cloud deployments. Cloud inference cost is often dominated by GPU compute time. Even a 1.3× speedup (achievable at ~50% sparsity with minimal accuracy impact) translates directly to 23% cost reduction—a substantial saving for services running at scale.

When to Prefer BLASST Over Alternatives

Prefer BLASST when:

  • You need to accelerate both prefill and decode phases (most production deployments)
  • You cannot retrain or fine-tune the model (using off-the-shelf checkpoints)
  • You want predictable, controllable sparsity (specify a target, get that sparsity)
  • You want minimal integration effort (single threshold parameter, existing API compatibility)
  • You are deploying on modern NVIDIA GPUs (Hopper or Blackwell with optimized kernels available)

Consider alternatives when:

  • You only need to optimize one phase (e.g., only decode-heavy workloads): KV cache compression methods like Quest or RocketKV may be sufficient
  • You can control the pretraining/fine-tuning process: methods like DSA or sparsity-aware training from scratch may achieve higher sparsity
  • You have very different memory hierarchy constraints: BLASST's kernel optimizations target NVIDIA GPU characteristics
  • You need guaranteed accuracy preservation: BLASST shows minimal degradation but is still approximate; dense attention is the only guarantee

For reproducibility, the artifact appendix (Appendix B) provides: - Docker environment setup scripts - Kernel benchmarking code - Automated threshold sweeps - Targeting H200 and B200 GPUs

The code is integrated into TensorRT-LLM and FlashInfer frameworks, available at the referenced GitHub repositories.