Skip to content

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

ArXiv: 2510.06499

🎯 Pitch

Webscale-RL introduces an automated pipeline that transforms massive, diverse pretraining text corpora into millions of high-quality, verifiable question–answer pairs for reinforcement learning. This innovation bridges the RL data bottleneck, enabling language models to attain strong benchmark performance and robust reasoning abilities with dramatically greater data efficiency—achieving continual pretraining results using up to 100× fewer tokens. By fundamentally scaling RL data to pretraining levels, Webscale-RL paves the way for more capable, adaptable, and affordable language model development.

1. Executive Summary

Webscale-RL introduces an automated data pipeline that converts large, general pretraining text corpora into millions of verifiable question–answer pairs suitable for reinforcement learning (RL). Trained on the resulting 1.2M-example, multi-domain dataset, a 3B-parameter model achieves broad benchmark gains and markedly higher token efficiency—matching continual pretraining performance with up to 100× fewer tokens (Figure 4; Sections 4–5).

2. Context and Motivation

  • Problem addressed
  • Modern large language models (LLMs) learn chiefly via imitation learning—next-token prediction during pretraining and supervised fine-tuning (SFT). This “teacher-forcing” setup means the model always sees the correct previous tokens at train time, but not at inference, which creates a training–generation gap and vulnerability to distribution shift (Section 1; Equation (1)).
  • RL can reduce this gap by optimizing the model’s own generations via reward feedback (Equation (2)), but RL for LLMs has been bottlenecked by data: existing RL datasets are orders of magnitude smaller and far less diverse than web-scale pretraining corpora (Figure 1; Section 1).
  • Why it matters
  • Practically, more robust reasoning and generalization require exposure to the model’s own generation dynamics; RL supplies this but needs abundant, verifiable data.
  • Theoretically and economically, if RL can achieve comparable or better performance with far fewer tokens, model development becomes more efficient (Section 1; Figure 4).
  • Prior approaches and limitations
  • Large-scale synthetic data pipelines exist for SFT and some RL, but they often:
    • Focus on narrow domains (especially math and code) and thus lack coverage of general knowledge (Table 1).
    • Rely on distillation from stronger teacher models, tying dataset ceiling to teacher capability (Section 4.2).
    • Are difficult to scale because they source queries from limited origins (Table 1; Section 4.2).
  • Positioning of this work
  • Webscale-RL directly converts pretraining documents into verifiable QA pairs suitable for RL, retaining the breadth of web-scale text. It introduces mechanisms for quality control and diversity (domain-specific demonstrations and persona-driven generation) and demonstrates both performance and token-efficiency advantages over continual pretraining and data-refinement pipelines (Sections 3–5).

3. Technical Approach

This section explains both the learning formulation and the data engine that makes RL at scale possible.

  • Pretraining vs. RL objectives
  • Pretraining objective (teacher-forcing imitation): minimize negative log-likelihood of the next token on a static dataset (Equation (1), Section 3.1). This teaches patterns from demonstrations but does not expose the model to errors it may make at inference.
  • RL objective: maximize expected reward over queries by generating answers online (Equation (2), Section 3.1). In this work, reward is binary: 1 if the model’s final answer matches the ground truth, 0 otherwise. Each training instance is therefore a verifiable QA pair.
  • Webscale-RL pipeline (Figure 2; Section 3.2)
  • Goal: Transform generic pretraining documents into high-quality, diverse, RL-ready QA pairs at scale.
  • Design principles
    • Diversity and domain coverage: retain the breadth of pretraining corpora.
    • Verifiability: ensure questions are self-contained and answers are short and checkable.
    • Leakage prevention: avoid trivial questions where the answer is embedded in the prompt.
  • Four stages 1) Data Filtering
    • Removes documents unlikely to yield verifiable QA pairs using heuristics and an LLM filter (Section 3.2).
    • Filters out non-informative boilerplate (e.g., navigation) and non-self-contained fragments. 2) Domain Classification and Persona Assignment
    • Classifies each document into domains (e.g., commerce, healthcare, social science) using an LLM-based classifier to pick relevant few-shot exemplars later (Section 3.2).
    • Assigns multiple personas—roles representing different viewpoints or information needs (e.g., “medical expert,” “patient,” “health journalist”) to induce diverse question styles from the same source (Section 3.2). Persona here is a prompt-time role specification, not a user profile. 3) Verifiable QA Generation
    • A QA generator (LLM) receives: the filtered document, its domain tag, assigned persona, and few-shot exemplars drawn from a domain-specific demonstration library (Section 3.2).
    • The question must be self-contained (include necessary context) because the RL-trained model will not see the source document at training time.
    • The answer must be short and verifiable (e.g., number, date, name, short phrase), which reduces generation complexity and enables reliable automatic checking (Section 3.2). 4) Quality Check and Leakage Control
    • An LLM verifier confirms: (a) answer correctness grounded in the source; (b) no info leakage—i.e., the question does not trivially reveal the answer (Section 3.2).
    • Final decontamination removes overlap with evaluation sets using lm-eval-harness (Section 3.2).
  • Implementation details
    • Prompt templates for each stage are provided in Appendix B.1.1.
    • The pipeline uses GPT-4.1 for filtering and QA generation, GPT-4.1-mini for classification and quality check (Section 4.1; Appendix B.1).
  • Dataset construction and properties
  • Sources include DCLM, Wikipedia, MegaMath, and Stack-v2; the final dataset has ~1.2M QA pairs across 9+ domains (Section 4.1; Table 3 in Appendix B.2).
  • Personas generate multiple QA pairs per document, broadening diversity (Appendix B.2.1 shows a concrete example).
  • Figure 3 (left) shows domain coverage; underrepresented areas in typical RL sets (e.g., Lifestyle >8.6%, Commerce >3.3%) are well covered.
  • Figure 3 (right) uses UMAP on Qwen3 embeddings to visualize question diversity: Webscale-RL is more evenly spread than Nemotron, which clusters around specific topics (Section 4.2).
  • Training and evaluation setup (Section 5.1; Tables 4–5)
  • Base model: Qwen2.5-3B.
  • RL algorithm: GRPO (Group Relative Policy Optimization), a PPO-style method that normalizes rewards within a group of sampled responses to stabilize updates (Section 2; Table 4).
  • Reward: binary exact-match against ground truth; an LLM is used to judge matches when needed (Sections 3.1 and B.3.2).
  • To avoid bias toward RL models’ better instruction-following, all non-RL baselines receive an additional SFT stage on a 10K curated set (Appendix B.3.1).
  • Baselines:
    • Continual Pretraining on the original pretraining corpus (1M documents) followed by SFT.
    • Advanced data-refinement pipelines (QuRating, ProX, Generative Data Refinement) applied to the same corpus, then continual pretraining + SFT (Section 5.1).
  • Benchmarks and protocols: MMLU-pro (5-shot), BigBench (0-shot), GPQA-diamond (0-shot), MATH500 (0-shot), GSM8K (8-shot), MBPP (0-shot), EvalPlus aggregate for code (0-shot). Evaluation uses lm-eval-harness, LightEval, and EvalPlus defaults (Section 5.1; Table 5).
  • Design choices and rationale
  • Short, verifiable answers lower checking cost and reduce ambiguity (Section 3.2).
  • Persona-driven prompts intentionally vary question styles, reflecting multiple real-world audiences and improving diversity (Section 3.2; Figure 3).
  • Domain-specific few-shot exemplars guide the generator toward high-quality, context-appropriate questions (Figure 2; Section 3.2).
  • Token-efficiency comparison counts RL “tokens” as those in the original documents used to generate the RL data, ensuring fairness with continual pretraining’s token accounting (Section 5.3).

4. Key Insights and Innovations

  • An end-to-end, scalable RL data engine grounded in pretraining corpora (Figure 2; Sections 3–4)
  • What’s new: Instead of hand-crafting RL tasks or distilling from teachers, the pipeline systematically converts generic web-scale text into verifiable QA pairs.
  • Why it matters: It attacks the core bottleneck—RL data scarcity and narrow coverage—without sacrificing diversity. Because both questions and answers are grounded in source documents, the process can scale with pretraining corpora (Table 1; Section 4.2).
  • Persona-driven, domain-aware QA synthesis that preserves web-scale diversity (Section 3.2; Figure 3)
  • What’s new: Multiple personas per document + domain-specific exemplars yield varied, audience-appropriate questions from the same source.
  • Why it matters: The question-embedding visualization (Figure 3 right) shows Webscale-RL covers the space more uniformly than Nemotron, suggesting broader generalization potential beyond math/code.
  • Verifiability-first design with automated leakage control (Section 3.2)
  • What’s new: Explicit multi-stage verification checks correctness against the source and filters questions that give away answers.
  • Why it matters: RL depends on reliable reward signals. Ensuring correctness and non-leakage improves training stability and prevents reward hacking.
  • Data-efficiency advantage of RL at scale (Section 5.3; Figure 4)
  • What’s new: The study quantifies scaling behavior using comparable token budgets and shows steep RL scaling curves.
  • Why it matters: Quote from Figure 4 discussion: “RL training with approximately 10M tokens attains similar performance to continual pretraining with 1B tokens,” i.e., ~100× fewer tokens. This is a substantial efficiency result with practical implications.

5. Experimental Analysis

  • Evaluation methodology (Section 5.1; Table 5)
  • Models: Qwen2.5-3B finetuned with GRPO on 150K sampled Webscale-RL QA pairs (after an SFT warmup of 10K examples). Baselines start from the same base model.
  • Datasets/benchmarks: General knowledge and reasoning (MMLU-pro, BigBench), STEM (GSM8K, MATH500, GPQA-diamond), coding (MBPP, EvalPlus).
  • Fairness measures: All continual-pretraining and data-refinement baselines receive the same 10K SFT to enhance instruction-following, reducing bias that might favor RL (Section 5.1).
  • Decontamination: Overlaps with evaluation are removed via lm-eval-harness (Section 3.2).
  • Main quantitative results (Table 2; Section 5.2)
  • Average score across all tasks:
    • Base Qwen2.5-3B: 47.6
    • Best non-RL baseline (GDR): 48.7
    • Webscale-RL (RL): 52.1
    • Gap to larger Qwen2.5-7B base shrinks from 10.6 points to 6.1 points on average.
  • Per-task highlights:
    • MMLU-pro: 43.7 (Webscale-RL) vs 39.9–40.0 (non-RL baselines) and 37.8 (base 3B).
    • BigBench: 48.3 (Webscale-RL) vs 44.9–46.0 (best non-RL).
    • GPQA-diamond: 23.2 (Webscale-RL) vs 20.8 (best non-RL equals base) and 20.8 (base 3B).
    • MATH500: 58.0 (Webscale-RL) vs ~44.0–44.6 (non-RL), approaching the 7B base at 60.8.
    • GSM8K: Modest gain 78.5 vs 77.4 (best non-RL), likely due to saturation (Section 5.2).
    • Coding (MBPP, EvalPlus): Similar to baselines (around 55–58), reflecting less coding data in sources (Section 5.2).
  • Quote (Section 5.2): “We observe an average improvement of 3.4 over the strongest baseline (GDR).”
  • Scaling and efficiency results (Figure 4; Section 5.3)
  • Setup: Compare RL on Webscale-RL versus continual pretraining on the original corpus at matched token budgets. RL tokens are counted as tokens in the source documents that generated the RL data (fair accounting).
  • Findings:
    • “RL training with approximately 10M tokens attains similar performance to continual pretraining with 1B tokens” on MMLU-pro.
    • At 100M tokens, RL gains +4.4 average points over the 3B base, while continual pretraining is near-flat (Figure 4, right).
    • RL curves rise more steeply as tokens increase across MMLU-pro, BigBench, and average.
  • Dataset analyses (Section 4.2; Table 1; Figure 3)
  • Table 1: Webscale-RL (1.2M RL QA pairs) spans multiple domains and is “High” in scalability because it is directly grounded in pretraining data, unlike competition/human-only datasets (low) or distillation-based ones constrained by query sources (medium).
  • Figure 3:
    • Left: domain distribution includes underrepresented areas (Lifestyle >8.6%, Commerce >3.3%).
    • Right: 2D UMAP of question embeddings shows Webscale-RL distributed broadly versus Nemotron’s concentration in clusters.
  • Do the experiments support the claims?
  • The results consistently show RL > continual pretraining and refined pretraining on general reasoning and knowledge tasks, with especially large gains on MATH500 (Table 2).
  • Efficiency analysis (Figure 4) provides direct evidence for the headline “up to 100× fewer tokens” claim.
  • The coding results are mixed—acknowledged as a function of domain imbalance (Section 6).
  • Ablations and robustness checks
  • The paper provides multi-stage quality checks and decontamination details (Sections 3.2, B.1, B.3), domain/persona analyses (Figure 3), and a fairness-oriented evaluation protocol (Section 5.1).
  • There is no dedicated ablation isolating personas, verification strictness, or few-shot library size; these would strengthen causal attribution.

6. Limitations and Trade-offs

  • Assumptions and dependencies
  • Reliance on strong LLMs (GPT-4.1 / 4.1-mini) for generation, classification, and verification (Section 4.1; Appendix B.1). This incurs cost and may affect reproducibility if API behavior changes, though code and resulting dataset are released.
  • Reward design is exact-match on short answers. This is simple and stable but may undervalue partially correct or semantically equivalent responses and restrict question types to those with concise answers (Section 3.1).
  • Coverage and domain balance
  • Coding coverage is relatively low in the current build, leading to smaller gains on programming benchmarks (Section 6).
  • Broader web-derived biases can propagate into the RL dataset; the pipeline filters low-quality/boilerplate but does not claim full bias mitigation.
  • Computational trade-offs
  • While RL is token-efficient in learning signal, it still requires RL-specific infrastructure and a reward-checking pass. The paper notes “a substantial extra inference cost” from the reward model as a bottleneck and suggests exploring more efficient reward models (Section 6; B.3.2).
  • Evaluation scope
  • Results are on a 3B base with comparison to a 7B base; effects on much larger models and longer training horizons are not presented here.
  • No ablation quantifies the marginal value of personas, few-shot libraries, or each verification stage.
  • Leakage and contamination
  • The pipeline includes leakage prevention and decontamination (Section 3.2), but perfect prevention is hard in practice; residual overlaps or paraphrase leakage remain possible.

7. Implications and Future Directions

  • Impact on the field
  • Demonstrates a viable path to scale RL to pretraining levels by transforming generic corpora into verifiable RL tasks (Figure 2; Section 6). This reframes RL for LLMs from a small, niche post-training step into a broad, scalable training regime.
  • Provides evidence that RL can be substantially more token-efficient than imitation learning at comparable budgets (Figure 4), potentially changing the cost calculus of future LLM training.
  • Follow-up research enabled
  • Reward modeling:
    • Develop lighter, faster reward checkers for short-answer equivalence and semantic normalization to reduce inference cost (Section 6).
    • Explore richer reward functions (partial credit, process rewards) while maintaining verifiability.
  • Data pipeline extensions:
    • Rebalance sources to boost underperforming domains (e.g., integrate repository-scale code to improve coding performance; Section 6).
    • Expand beyond single-turn QA to multi-turn, tool-use, and function-calling tasks by adapting verification strategies (related to [39, 50] cited in Section 3.2).
  • Ablations and causal analysis:
    • Quantify the contribution of personas, domain-specific exemplars, and each verification stage to final performance.
  • Scaling studies:
    • Apply the pipeline to larger base models and longer RL horizons (e.g., “prolonged RL”), and analyze emergent capabilities (Section 2 references).
  • Practical applications
  • Building general-purpose assistants with stronger reasoning across diverse real-world domains—not just math/code—thanks to the dataset’s breadth (Figure 3).
  • Enterprise and domain-specific LLMs can adopt the pipeline to transform their proprietary corpora into verifiable RL data, potentially improving accuracy and robustness with fewer tokens than conventional continual pretraining.

Core takeaway: By turning the vast, heterogeneous pretraining web into millions of verifiable, persona-diverse RL questions (1.2M across 9+ domains; Section 4), Webscale-RL makes RL both scalable and efficient. Empirically, it improves a 3B model’s broad capabilities and achieves up to 100× token efficiency over continual pretraining (Table 2, Figure 4), offering a concrete path to align RL’s promise with pretraining-scale practice.