Transformers are Universal In-context Learners¶

🎯 Pitch¶

This paper proves that deep transformers are universal approximators of continuous in‑context mappings—functions mapping a token to an output conditioned on an arbitrarily large (even infinite) context—by modeling contexts as probability measures and using Wasserstein continuity. Crucially, the construction yields a single transformer that achieves any fixed precision without increasing embedding dimension or number of attention heads with context size, providing a principled foundation for transformers’ in‑context computation and mean‑field limits across NLP and vision applications.

1. Executive Summary (2-3 sentences)¶

This paper gives a rigorous universal approximation theorem showing that deep transformer architectures can approximate any continuous “in-context mapping”—a function that maps a token to an output while depending on a context of other tokens—even when the context contains an arbitrary (including infinite) number of tokens (Theorem 1, Theorem 2). The key significance is that, for a fixed target precision \(\varepsilon\), the construction uses a single transformer architecture whose embedding dimension and number of attention heads do not grow with \(\varepsilon\), and the same model definition applies uniformly to finite-token and infinite-token (“mean-field”) contexts via a probability-measure formalism.

2. Context and Motivation¶

What specific problem or gap does this paper address?
Standard universality results for neural networks typically assume fixed-size inputs in Euclidean space (e.g., an \(n\)-token sequence treated as a vector in \(\mathbb{R}^{d\times n}\)).
Transformers in practice handle variable and potentially very long contexts, and the paper targets a mathematically uniform expressivity statement that does not depend on the number of tokens \(n\).
The core gap: prior transformer universality results often require architectural quantities to scale with token count or precision (the paper explicitly contrasts with results needing embedding dimension growth with token count).
Why is this problem important?
If one wants a theory of “in-context learning” that matches how transformers are used (prompts of varying length; potentially huge sets of patches; and even conceptual limits with infinitely many tokens), one needs a notion of input space where “context size” can vary smoothly and still permit approximation theorems.
The paper’s approach formalizes “contexts” as probability measures over token embeddings, so one can compare contexts of different sizes and define continuity in a principled topology (weak\(^\*\) / Wasserstein).
What prior approaches existed, and where do they fall short (as positioned here)?
The paper cites a detailed universality account (not reproduced here) that uses shallow transformers with few heads but requires embedding dimension to grow with the number of tokens (discussion around “Universality of transformers” and the comparison in Section 1).
It also positions itself relative to work modeling attention over probability distributions and studying smoothness/Lipschitz behavior of attention layers (Section 1; and the measure-theoretic formulation in Section 2).
How does this paper position itself relative to existing work?
It emphasizes deep transformer compositions and a measure-theoretic view of attention to obtain a universality statement that:
- works uniformly over all token counts, including infinite-token limits (Section 2.2, Theorem 1),
- keeps embedding dimension fixed (independent of \(\varepsilon\)) and keeps the number of heads fixed up to proportionality with output dimension (Theorem 1, Theorem 2),
- handles both unmasked (bidirectional) and masked causal attention, with extra regularity assumptions in the causal case (Section 4.1).

3. Technical Approach¶

3.1 Reader orientation (approachable technical breakdown)¶

The system is a mathematical transformer model re-expressed as an operator that takes (i) a context and (ii) a query token and outputs an updated token.
It solves the problem of showing universal approximation of “in-context functions” in a setting where the context may have arbitrarily many tokens, by representing the context as a probability distribution over token embeddings and proving density results via Stone–Weierstrass.

3.2 Big-picture architecture (diagram in words)¶

Input: a context (either a finite token ensemble or a probability measure over tokens) and a token \(x\) (and in causal settings, also a time \(t\)).
Core components:
A measure-theoretic multi-head attention map \(\Gamma_\theta(\mu, x)\) that updates \(x\) using weighted averages over the distribution \(\mu\) (Eq. (9) unmasked; Eq. (12) masked).
A context-free MLP map \(F_\xi(x)\) applied to each token independently (Eq. (4), Eq. (11)).
Layer composition defined by a special “in-context composition” operator \(\diamond\) that updates the context measure by push-forward after each layer (Eq. (10) unmasked; Eq. (13) masked).
Output: an approximation to a target in-context map \(\Lambda^\star(\mu, x)\) (or \(\Lambda^\star(\mu, x, t)\) in the causal setting) uniformly over compact domains.

3.3 Roadmap for the deep dive¶

First, define “in-context mappings” for classical finite-token attention and rewrite attention as token-wise functions \(G_\theta(X,\cdot)\) (Eq. (2), Eq. (3)).
Next, lift finite token sets to probability measures and define measure-theoretic attention \(\Gamma_\theta(\mu,\cdot)\) that makes sense for any token count (Eq. (8)–(11)).
Then, prove universality in the unmasked case (Theorem 1) by:
building an algebra of “elementary” attention-based scalar functions (Eq. (14)),
proving this algebra is dense via Stone–Weierstrass (Proposition 1),
lifting from scalars to vectors and from products to deep transformer compositions (Lemmas 2–3).
Finally, address masked (causal) attention by:
adding a space–time lifting and “masked context” measures \(\mu_t\) (Eq. (12), Definition 2),
restricting to Lipschitz-in-time contexts to regain compactness (Definition 1, Lemma 5),
reducing causal identifiable maps to a form amenable to the same approximation strategy (Definition 4, Lemma 4, Theorem 2).

3.4 Detailed, sentence-based technical breakdown¶

Framing (type of paper + core idea).
This is a theoretical approximation-theory paper whose core idea is to model transformer attention as an operator on probability measures of tokens, so that contexts of varying (even infinite) size live in a single compact topological space, enabling a Stone–Weierstrass universality proof for deep transformer compositions (Section 2, Section 3, Section 4).
Step 1: Start from standard finite-token attention and rewrite it as an “in-context function.”
A finite token ensemble is written as \(X=(x_i)_{i=1}^n \in \mathbb{R}^{d_{\text{in}}\times n}\).
A single attention head is defined (classically) using key/query/value matrices and a row-wise SoftMax; the paper writes it as [ \mathrm{Att}\theta(X) := V X \,\mathrm{SoftMax}(X^\top Q^\top K X / \sqrt{k}) \in \mathbb{R}^{d. ]}}\times n
A multi-head attention block adds a residual/skip connection: [ \mathrm{MAtt}\theta(X) := X + \sum ]}^H W_h\, \mathrm{Att}_{\theta_h}(X) \quad \text{(Eq. (1)).
In the unmasked case, \(\mathrm{MAtt}_\theta\) can be rewritten token-wise as applying the same “in-context” function to each token: [ \mathrm{MAtt}\theta(X) = (G\theta(X, x_i))_{i=1}^n, ] where \(G_\theta(X,x)\) is an explicit weighted sum over all tokens (Eq. (2)).
In the masked (causal) case, the update depends on the token index \(i\) because attention only looks backward (\(j \le i\)), yielding \(G_\theta(X,x,i)\) (Eq. (3)).
Step 2: Define transformer depth as composition in the “in-context function” viewpoint.
A transformer (ignoring normalization) alternates attention blocks and token-wise MLP blocks: [ \mathrm{MLP}{\xi_L}\circ \mathrm{MAtt}}\circ \cdots \circ \mathrm{MLP{\xi_1}\circ \mathrm{MAtt} ]} \quad \text{(Eq. (4)).
MLP blocks act independently per token: \(\mathrm{MLP}_\xi(X) = (F_\xi(x_i))_{i=1}^n\).
The paper defines an explicit “in-context composition” operator \(\diamond\) that tracks the fact that the context itself changes layer-by-layer (Eq. (5) unmasked; Eq. (6) masked), leading to the in-context composition expression (Eq. (7)).
Step 3: Replace variable-length token lists by a single object: a probability measure over token embeddings.
For a compact token domain \(\Omega \subset \mathbb{R}^d\), contexts are treated as probability measures \(\mu \in \mathcal{P}(\Omega)\).
A finite set of \(n\) tokens is embedded as the empirical measure [ \mu = \frac{1}{n}\sum_{i=1}^n \delta_{x_i} \quad \text{(Eq. (8)).} ]
This representation is what lets the theory compare contexts with different \(n\), and it ties smoothness/continuity to weak\(^\*\) topology (and metrization via Wasserstein distance, reviewed in Appendix A).
Step 4: Define measure-theoretic attention \(\Gamma_\theta(\mu,x)\) that works for finite or infinite contexts.
In the unmasked setting, the in-context attention update becomes an integral operator: [ \Gamma_\theta(\mu, x) := x + \sum_{h=1}^H W_h \int \frac{\exp!\left(\sqrt{\frac{1}{k}}\langle Q_h x, K_h y\rangle\right)}{\int \exp!\left(\sqrt{\frac{1}{k}}\langle Q_h x, K_h z\rangle\right)\,d\mu(z)} \,V_h y \, d\mu(y) \quad \text{(Eq. (9)).} ]
The key identity is that for an empirical measure built from \(X\), this recovers the discrete attention map: \(G_\theta(X,x)=\Gamma_\theta(\mu,x)\) (Section 2.2).
Each layer also induces a transformation of the context measure by push-forward: [ \mu \mapsto \Gamma_\theta(\mu)_# \mu, ] meaning tokens get “moved” by the map and the distribution moves with them (Appendix A, Eq. (22)).
Composition in the measure setting explicitly updates the intermediate measure: [ (\Gamma_2 \diamond \Gamma_1)(\mu,x) := \Gamma_2(\mu_1, \Gamma_1(\mu,x)),\quad \mu_1 := \Gamma_1(\mu)_#\mu \quad \text{(Eq. (10)).} ]
A deep transformer over measures is then written as an alternating composition [ F_{\xi_L}\diamond \Gamma_{\theta_L}\diamond \cdots \diamond F_{\xi_1}\diamond \Gamma_{\theta_1} \quad \text{(Eq. (11)).} ]
Step 5: Prove universality in the unmasked case (Theorem 1) via Stone–Weierstrass.
Goal (Theorem 1). For any compact \(\Omega\subset\mathbb{R}^d\) and any continuous target map [ \Lambda^\star:\mathcal{P}(\Omega)\times \Omega \to \mathbb{R}^{d'}, ] the paper constructs a deep transformer such that for every \(\varepsilon>0\), [ \sup_{(\mu,x)\in \mathcal{P}(\Omega)\times\Omega}\left\lVert \big(F_{\xi_L}\diamond \Gamma_{\theta_L}\diamond \cdots \diamond F_{\xi_1}\diamond \Gamma_{\theta_1}\big)(\mu,x) - \Lambda^\star(\mu,x)\right\rVert \le \varepsilon, ] with architectural controls:
> \(\ d_{\text{in}}(\theta_\ell)\le d+3d',\ \ d_{\text{head}}(\theta_\ell)=k(\theta_\ell)=1,\ \ H(\theta_\ell)\le d'\) (Theorem 1).
Elementary building blocks. The proof defines a scalar “elementary” in-context mapping [ \gamma_\lambda(\mu,x) := \langle x,a\rangle + b + \int \frac{e^{c(\langle x,a\rangle+b)(\langle y,a\rangle+b)}\, v(\langle a,y\rangle+b)}{\int e^{c(\langle x,a\rangle+b)(\langle z,a\rangle+b)}\, d\mu(z)}\, d\mu(y) \quad \text{(Eq. (14)).} ] Mechanistically, this is realized by composing: 1) an affine scalar MLP \(F_\xi(x)=\langle a,x\rangle+b\), and
2) a single-head (1D) attention with skip connection (Section 3.2).
Algebra + density. The paper defines an algebra \(\mathcal{A}\) generated by finite sums of finite products of such \(\gamma_\lambda\) functions (Section 3.2) and proves: > Proposition 1: \(\mathcal{A}\) is dense in the space of continuous functions on \(\mathcal{P}(\Omega)\times\Omega\) under the (weak\(^\*\times \ell^2\)) topology.
Why Stone–Weierstrass applies. The argument checks the usual conditions: 1) compactness of \(\mathcal{P}(\Omega)\times\Omega\) (citing a compactness result; Proposition 1 proof sketch),
2) constants are included (choose parameters so \(\gamma_\lambda\equiv 1\)), and
3) point-separation: if all \(\gamma_\lambda(\mu,x)=\gamma_\lambda(\mu',x')\) then \((\mu,x)=(\mu',x')\) (Eq. (15)).
Key technical identifiability lemma (injectivity). To separate measures, the proof reduces equality of \(\gamma_\lambda\) values to equality of a generalized Laplace-like transform [ L(\mu)(a,c):=\frac{\int e^{c\langle a,y\rangle}\langle a,y\rangle\, d\mu(y)}{\int e^{c\langle a,z\rangle}\, d\mu(z)} \quad \text{(Eq. (16))} ] and shows: > Lemma 1: \(\mu \mapsto L(\mu)\) is injective (Lemma 1; details in Appendix B.1 via 1D moment recovery + projection/Radon-transform reasoning).
Lift from scalars to vectors. Lemma 2 explains how to approximate each output coordinate and combine them into a vector-valued approximation using attention heads arranged per coordinate (Lemma 2, Eq. (17)).
Replace explicit products by deep composition. The approximation algebra involves products (component-wise multiplications). Lemma 3 shows how to approximate these multiplication operations with MLPs and realize the resulting computation as a deep transformer, maintaining the same dimension/head bounds stated in Theorem 1 (Lemmas 3, and the constructive machinery in Appendix B.3, including Lemmas 8–9).
A concrete “system/data pipeline diagram in words” (what happens first, second, third).
First, encode the context tokens as a probability measure \(\mu\) (empirical if tokens are finite, Eq. (8)).
Second, apply a layer \(\Gamma_{\theta_1}\) that maps \((\mu,x)\mapsto \Gamma_{\theta_1}(\mu,x)\) and simultaneously pushes the context forward to \(\mu_1=\Gamma_{\theta_1}(\mu)_\#\mu\) (Eq. (10)).
Third, apply a token-wise MLP \(F_{\xi_1}\) to the updated token (and, implicitly through the push-forward formalism, to context tokens).
Then, repeat for \(L\) layers, always updating the context measure via push-forward after attention, and composing token updates via \(\diamond\) (Eq. (11)).
Finally, the output token approximates \(\Lambda^\star(\mu,x)\) uniformly over all \(\mu\in\mathcal{P}(\Omega)\) and \(x\in\Omega\) (Theorem 1).
Step 6: Handle masked (causal) attention by space–time lifting and restricting contexts to be Lipschitz in time.
In causal attention, permutation equivariance is broken by the mask, so the paper augments each token with a time coordinate \(t\in[0,1]\) and treats the context as a space–time measure \(\mu \in \mathcal{P}(\Omega\times[0,1])\) (Section 2.3).
Masked attention becomes an integral restricted to times \(s\le t\): [ \Gamma_\theta(\mu,x,t)=x+\sum_{h=1}^H W_h\int \frac{\exp(\cdots)\,\mathbf{1}{[0,t]}(r)}{\int \exp(\cdots)\,\mathbf{1} V_h y\, d\mu(y,r) \quad \text{(Eq. (12)).} ]}(s)\,d\mu(z,s)
Composition updates only the spatial component while keeping time fixed: [ (\Gamma_2\diamond \Gamma_1)(\mu,x,t):=\Gamma_2(\mu_1,\Gamma_1(\mu,x,t),t),\quad \mu_1:= (\Gamma_1(\mu),\mathrm{Id}{\mathbb{R}})#\mu \quad \text{(Eq. (13)).} ]
Because masking can introduce irregular dependence on \(t\), the paper restricts contexts to a compact class:
- Lipschitz contexts: the conditional distribution \(t\mapsto \mu(\cdot\mid t)\) must be \(C\)-Lipschitz in Wasserstein-\(W_2\) (Definition 1).
- A technical condition \(\bar\mu(\{0\})\ge \sigma\) is imposed to ensure masked normalization is well-defined at early times (Definition 1, Definition 2; and Remark 2 discusses alternatives).
Define the masked measure (normalized restriction to \([0,t]\)): [ \mu_t := \frac{\mathbf{1}_{[0,t]}}{\bar\mu([0,t])}\cdot \mu \quad \text{(Eq. (19), Definition 2),} ] with continuity in \(t\) established (Lemma 10).
Define the target class of functions to approximate:
- Causality: \(\Lambda(\mu,x,t)=\Lambda(\mu_t,x,t)\) (Definition 3, Eq. (20)).
- Identifiability: if \(\mu_t=\mu_{t'}\) then the mapping’s behavior is consistent across those times (Eq. (21)); the paper emphasizes this is necessary because transformers themselves yield identifiable maps (Remark 1, Lemma 13).
Step 7: Prove universality in the masked case by reducing to an unmasked-style problem (Theorem 2).
Theorem 2 mirrors Theorem 1 but on the restricted domain \(\mathrm{Lip}^\sigma_C(\tilde\Omega)\) and for continuous causal identifiable maps.
The proof introduces a reduced mapping \(\bar\Lambda\) that converts the problem of approximating \(\Lambda(\mu,x,t)\) into approximating a function of \((\mu_t,x)\) (Definition 4, Lemma 4).
The set [ X^\sigma_C := {(\mu_t,x): \mu\in \mathrm{Lip}^\sigma_C(\tilde\Omega),\ x\in\Omega,\ t\in[0,1]} ] is shown compact (Lemma 5), which is the key property needed to re-run Stone–Weierstrass.
Once reduced, the approximation strategy is “basically the same” as the unmasked case (Proposition 2), yielding the same architectural size bounds as Theorem 1 (Theorem 2 statement).
Core configurations and “hyperparameters” (as stated in the theorems; no training setup is given in the provided content).
For both Theorem 1 and Theorem 2, the constructed transformers satisfy:
- Attention head output dimension: dhead(θℓ) = 1 (scalar head outputs).
- Key/query dimension: k(θℓ) = 1.
- Number of heads: H(θℓ) ≤ d′ (scales with output dimension).
- Token embedding dimension through layers: din(θℓ) ≤ d + 3d′ (depends on input and output dimension, not on precision \(\varepsilon\)).
The paper explicitly notes it does not provide quantitative bounds on:
- the required depth \(L\) as a function of \(\varepsilon\),
- the magnitude growth of tokens across layers,
- or the number/size of MLP parameters needed for multiplication/squaring approximations (discussion after Theorem 1; Remark 3 in Appendix B.3).
Worked micro-example (illustrative walk-through consistent with the paper’s formalism).
Suppose the “context” is two tokens \(x_1,x_2\in\Omega\subset\mathbb{R}^d\) and we form the empirical measure \(\mu=\tfrac12(\delta_{x_1}+\delta_{x_2})\) (Eq. (8)).
A single unmasked measure-theoretic attention layer \(\Gamma_\theta\) (Eq. (9)) computes, for a query token \(x\): 1) scores \(s_h(y)=\exp(\sqrt{1/k}\langle Q_h x, K_h y\rangle)\) for \(y\in\{x_1,x_2\}\),
2) a normalization \(Z_h(x)=\int s_h(z)\,d\mu(z)=\tfrac12(s_h(x_1)+s_h(x_2))\), and
3) an update \(u_h(x)=\int \frac{s_h(y)}{Z_h(x)}\,V_h y\,d\mu(y)\), which becomes a weighted average of \(V_h x_1\) and \(V_h x_2\).
The residual connection adds \(x+\sum_h W_h u_h(x)\), making \(\Gamma_\theta(\mu,x)\) a context-dependent map of \(x\).
If we now stack layers using \(\diamond\) (Eq. (10)), the context itself updates as \(\mu\mapsto \mu_1=\Gamma_{\theta_1}(\mu)_\#\mu\), meaning that in layer 2 the attention integrates over the transformed tokens from layer 1, matching the “tokens evolve through layers” intuition but stated as measure push-forwards.

4. Key Insights and Innovations¶

(1) Universality that is uniform over context length (including infinite contexts).
The central innovation is the measure-based formulation (Eq. (8)–(11)), which allows a single approximation statement to cover any number of tokens.
Theorem 1 explicitly asserts the approximating transformer operates “independently of \(n\) (it even works for an infinite number of tokens)” and does not need dimension growth with \(n\).
(2) Fixed embedding dimension and head count independent of precision \(\varepsilon\).
Theorem 1 (and Theorem 2) provide architectural bounds where \(d_{\text{in}}(\theta_\ell)\le d+3d'\) and \(H(\theta_\ell)\le d'\), with no dependence on \(\varepsilon\) in these quantities.
This contrasts with universality styles that trade approximation accuracy for width growth; here, the cost of accuracy is pushed into depth/MLP complexity (without quantitative bounds).
(3) A Stone–Weierstrass strategy specialized to attention-defined function classes.
The proof is not “MLPs are universal, therefore transformers are universal,” but instead constructs an algebra generated by attention+affine blocks (Eq. (14)) and proves it is dense (Proposition 1).
The injectivity lemma (Lemma 1 via Eq. (16)) is the technical hinge that ensures the elementary attention features can separate measures.
(4) A principled causal (masked) extension using space–time measures and identifiability.
The causal case is handled by lifting tokens to \((x,t)\) and using masked measures \(\mu_t\) (Eq. (12), Eq. (19)).
The paper identifies two necessary structural restrictions for masked universality:
- Lipschitz-in-time contexts for compactness (Definition 1, Lemma 5),
- Identifiability as a sharp requirement because transformer-induced maps are identifiable and uniform limits preserve identifiability (Remark 1, Lemma 13).

5. Experimental Analysis¶

Evaluation methodology, datasets, metrics, baselines, setup:
The provided paper content contains no empirical experiments, datasets, training runs, or quantitative benchmarks. The work is entirely theoretical, with results stated as approximation theorems (Theorem 1 and Theorem 2) and supporting lemmas/propositions.
Main quantitative results with specific numbers:
There are no performance numbers (e.g., accuracy, loss, BLEU, etc.) because no experiments are reported.
The only “numerical” aspects are architectural bounds in the theorems (e.g., \(d_{\text{head}}=1\), \(k=1\), \(H\le d'\), and \(d_{\text{in}}\le d+3d'\)).
Do the results convincingly support the claims (within the paper’s scope)?
For the claim “transformers are universal in-context learners” in the stated formal sense, the paper provides:
- explicit definitions of the function class being approximated (continuous maps on \(\mathcal{P}(\Omega)\times\Omega\); and continuous causal identifiable maps on \(\mathrm{Lip}^\sigma_C(\tilde\Omega)\times\tilde\Omega\)),
- a density argument (Proposition 1) and a constructive realization via deep compositions (Lemmas 2–3; Appendix B.3’s constructive steps).
The proofs are existential and non-quantitative, and the paper itself flags that limitation (discussion after Theorem 1; Conclusion).
Ablations/failure cases/robustness checks:
Not applicable in an experimental sense. Theoretical “sharpness” is discussed for identifiability and Lipschitz assumptions in the masked case (Remark 1).

6. Limitations and Trade-offs¶

Non-quantitative universality (no rates).
The paper explicitly states a weakness: it provides no explicit control over how the number of layers \(L\) or MLP parameter complexity depends on \(\varepsilon\) (discussion after Theorem 1; Remark 3).
As a result, the theorem guarantees approximation exists but does not tell you how large the network must be.
Head/output-dimension trade-off in the construction.
Each head is scalar (\(d_{\text{head}}=1\)), and the number of heads scales as \(H\le d'\) (Theorem 1/2).
The paper notes improving the balance between head dimension and number of heads is an open problem (discussion after Theorem 1).
Potential uncontrolled token magnitude growth across layers.
The construction does not provide an a priori bound on how token values can grow through the layers, which matters if one wanted quantitative approximation rates for MLP multiplication approximations (discussion after Theorem 1; Remark 3 uses this point to explain why quantitative bounds are hard).
Masked (causal) case requires extra assumptions that restrict the domain.
Approximation is restricted to contexts in \(\mathrm{Lip}^\sigma_C(\tilde\Omega)\) (Definition 1), meaning the conditional token distribution must vary Lipschitzly in time in \(W_2\).
The time marginal constraint \(\bar\mu(\{0\})\ge \sigma\) excludes some continuous-time marginals (Remark 2), although the paper suggests an alternative of fixing the time marginal instead.
Identifiability is a hard constraint in the masked theorem.
Theorem 2 requires the target map to be identifiable (Definition 3), and Remark 1 + Lemma 13 argue this is sharp because identifiable maps can only converge uniformly to identifiable maps.

7. Implications and Future Directions¶

How this changes the landscape (within the formal setting of the paper).
The results provide a clean theoretical statement that, under continuity assumptions on measure-conditioned mappings, deep transformer-style attention+MLP stacks are expressive enough to approximate any such mapping uniformly, while being agnostic to context length (Theorem 1).
It reframes transformer expressivity as approximation on spaces of probability measures rather than on fixed-length sequences, which is aligned with variable-length contexts and “mean-field” (infinite-token) interpretations (Section 2.2).
Follow-up research directions suggested by the paper’s own discussion.
Quantitative approximation bounds: The conclusion suggests that imposing smoothness using distances like Wasserstein could lead to explicit rates (Conclusion).
Reduce head scaling with output dimension: The paper highlights that \(H\propto d'\) arises from the proof technique and that lowering this dependence likely requires new ideas beyond Stone–Weierstrass (Conclusion and post-Theorem-1 discussion).
Relax masked-case marginal constraints: Remark 2 sketches that fixing the time marginal could replace the \(\bar\mu(\{0\})\ge\sigma\) assumption with “minor modifications.”
Practical applications / downstream use cases (conceptual, given no experiments).
The theorems are not performance claims, but they conceptually support the idea that transformers can represent very general computations “in-context,” including in regimes where context size varies or becomes extremely large, provided the target behavior depends continuously on the token distribution (Theorem 1).
The masked result conceptually characterizes what kinds of causal behaviors are representable/approximable when you demand uniform approximation over a class of time-varying contexts (Theorem 2).
Repro/Integration Guidance: when to prefer this viewpoint (within the paper’s scope).
Prefer the measure-theoretic formulation when the key modeling question depends on handling variable-length contexts in a single mathematical space, or when you want to reason about limits where the context becomes very large or continuous (Section 2.2).
In causal problems, the paper’s results suggest that if you want uniform approximation guarantees, you may need to restrict attention to contexts that vary regularly in time (Definition 1) and to target maps that satisfy identifiability (Definition 3), because those properties are structurally built into masked attention compositions (Lemma 11, Lemma 13).