Skip to content

KIMI K1.5: SCALING REINFORCEMENT LEARNING WITH LLMS

ArXiv: 2501.12599

🎯 Pitch

This technical report introduces Kimi k1.5, a multimodal LLM trained with a deliberately simple RL pipeline that scales chain-of-thought context to 128k tokens and applies robust policy-optimization and infrastructure techniques (partial rollouts, length penalty, curriculum/prioritized sampling) instead of complex planning machinery. By turning long-context RL into practical training recipes and long→short transfer methods, the work yields state-of-the-art reasoning across math, code, and vision benchmarks and opens a new, data‑amplifying axis for scaling model capabilities.

1. Executive Summary (2-3 sentences)

Kimi k1.5 is a multimodal large language model trained with a deliberately “simple” reinforcement learning (RL) setup that emphasizes scaling the RL context window to 128k tokens and improving policy optimization rather than adding complex planning machinery like Monte Carlo tree search or value functions (Introduction; Section 2.3; Section 2.6). The report’s significance is practical: it describes a full training recipe (data curation → long-CoT warmup → RL → infrastructure) and shows strong reasoning results across math, code, and vision benchmarks (Figures 1–2; Tables 2–3), while also proposing long2short methods to transfer long chain-of-thought gains into shorter, cheaper outputs (Section 2.4; Figure 7).

2. Context and Motivation

  • Problem/gap addressed
  • Standard LLM scaling via next-token pretraining is ultimately constrained by the supply of high-quality training data (Introduction).

  • RL could, in principle, create a new scaling axis by letting the model “generate” training experience through exploration and reward feedback, but the report claims prior published work has not been competitive (Abstract; Introduction).

  • Why this matters

  • The target capability is complex reasoning (math problem solving, programming, and multimodal reasoning over images+text), where “just predicting the next token” may not push systematic planning behaviors as effectively (Abstract; Section 2.3.1).

  • Prior approaches and shortcomings (as positioned here)

  • At inference time, “planning” methods build explicit search trees over intermediate steps (the report frames this broadly as tree search guided by a critic/value estimator) (Section 2.3.1).
  • These planning methods require additional components (critic/value estimation, search procedure, potentially parallel computation), which complicates deployment and/or training (Section 2.3.1).

  • The report’s position: with sufficiently long context and RL training, an autoregressive model can learn to perform an implicit search (planning/reflection/correction) without explicit search trees or value functions (Introduction; Section 2.3.1; “Simplistic Framework” bullet in Introduction).

  • How the report positions its approach

  • Core claim: long-context RL (up to 128k) + improved policy optimization + infrastructure (partial rollouts) is enough to get strong long-CoT behavior and strong benchmark performance, while remaining “simplistic” (Introduction; Sections 2.3, 2.6; Figures 5–6).

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

  • The system is a multimodal LLM training pipeline that uses supervised fine-tuning (SFT) plus reinforcement learning to teach the model to solve verifiable reasoning tasks with long chains of thought.
  • It solves the problem of improving reasoning by (i) training on curated prompts with objective verifiers/reward models, (ii) optimizing a KL-regularized RL objective with an off-policy mirror-descent-style update, and (iii) scaling the usable reasoning context via a distributed rollout/training infrastructure (Sections 2.1–2.6).

3.2 Big-picture architecture (diagram in words)

  • Data & prompt sources → RL prompt set curation (diversity/difficulty/evaluability, anti-hacking filters) (Section 2.1)
    → Long-CoT warmup SFT dataset (verified long reasoning traces) (Section 2.2)
    → Iterative RL loop:
  • Rollout workers sample long CoT trajectories from the current policy.
  • Reward models / verifiers score outputs (code execution, math RM, vision/text verifiers).
  • Replay buffer stores trajectories (including partial segments).
  • Trainer workers update the policy using the chosen RL optimizer and sampling strategy.
  • Hybrid deployment alternates training (Megatron) and inference/rollouts (vLLM) with fast weight transfer (Figure 3; Figure 4; Section 2.6).
  • Optional long2short stage compresses long-CoT ability into shorter outputs using merging / rejection sampling / DPO / RL with length penalties (Section 2.4; Figure 7).

3.3 Roadmap for the deep dive

  • First, define the RL problem formulation for chain-of-thought generation (Section 2.3.1; Eq. (1)).
  • Second, explain the policy optimization algorithm (KL-regularized mirror descent variant, surrogate loss, gradient form) (Section 2.3.2; Eqs. (2)–(3)).
  • Third, cover token-length control (length penalty) and sampling strategies (curriculum + prioritization) because they change what data the optimizer sees (Sections 2.3.3–2.3.4).
  • Fourth, explain reward/verification pipelines for code, math, and vision because RL quality depends on evaluator correctness (Section 2.3.5).
  • Fifth, describe the infrastructure (partial rollouts, hybrid Megatron↔vLLM deployment, sandbox) that makes 128k-context RL practical (Section 2.6; Figures 3–4).
  • Finally, explain long2short methods as an add-on stage for token efficiency (Section 2.4; Figure 7).

3.4 Detailed, sentence-based technical breakdown

Framing: This is primarily an empirical systems + algorithmic training recipe paper: it combines an RL objective for long-CoT LLMs, a specific policy optimization approach, and a distributed training/serving infrastructure that enables long-context RL at scale (Sections 2.3 and 2.6).

3.4.1 RL problem setting: reasoning as sampling a chain of thought

  • The training data is described as a set of problems and ground-truth answers:
    [ D={(x_i, y_i^*)}_{i=1}^n ] where x is a prompt/problem and y* is the correct answer (Section 2.3.1).
  • The model is a policy \(\pi_\theta\) that generates:
  • an ordered sequence of intermediate “thoughts” \(z=(z_1,\dots,z_m)\), and then
  • a final answer \(y\), all as tokens sampled autoregressively (Section 2.3.1).
  • The report’s conceptual move is to relate explicit planning/search to “flattened context.” It describes classical planning as building a search tree of partial solutions and scoring nodes with a critic/value estimate, then repeatedly expanding promising nodes (Section 2.3.1). It then argues that if the full search history fits in context, a model could learn to implicitly approximate this process through autoregressive continuation conditioned on the whole history (Section 2.3.1).
  • Reward signal: For each generated answer \(y\), a reward \(r(x,y,y^*)\in\{0,1\}\) indicates correctness:
  • For verifiable tasks, reward is determined by predefined rules (e.g., code passes tests).
  • For freer-form ground truth, they train a reward model \(r(\cdot)\) (Section 2.3.1).
  • The basic RL objective is the expected reward over dataset problems and model samples:

    [ \max_\theta \ \mathbb{E}_{(x,y^)\sim D,(y,z)\sim \pi_\theta}[r(x,y,y^)] ]
    (Eq. (1), Section 2.3.1)

Plain-language paraphrase of Eq. (1): sample a problem, let the model “think” (generate a long reasoning trace) and answer, then update the model to increase the probability of outputs that receive a correctness reward.

3.4.2 Policy optimization: KL-regularized mirror-descent-style updates (and why)

  • The training proceeds in iterations. At iteration \(i\), the current policy \(\pi_{\theta_i}\) is treated as a reference policy (Section 2.3.2).
  • They optimize a KL-regularized objective:

    [ \max_{\theta} \ \mathbb{E}[r(x,y,y^*)] - \tau \, KL(\pi_\theta(x)\,|\,\pi_{\theta_i}(x)) ] (Eq. (2), Section 2.3.2)

Plain-language paraphrase of Eq. (2): improve reward, but penalize drifting too far from the previous model (controlled by \(\tau>0\)), which is intended to stabilize optimization.

  • The report states this objective has a closed-form optimal policy:

    [ \pi^(y,z|x)=\pi_{\theta_i}(y,z|x)\frac{\exp(r(x,y,y^)/\tau)}{Z} ] (Section 2.3.2, immediately after Eq. (2))

Interpretation: this is a Boltzmann reweighting of the old policy: correct samples (\(r=1\)) get upweighted relative to incorrect ones (\(r=0\)), with \(\tau\) controlling how sharp the preference is.

  • To make learning practical, the report introduces a surrogate loss based on a constraint derived from the closed form, and approximates the normalization term \(Z\) using samples from \(\pi_{\theta_i}\) (Section 2.3.2).
  • They further simplify in practice by using the empirical mean reward \(\bar r\) over samples as an effective approximation (Section 2.3.2).
  • The final gradient form they present is:

    [ \frac{1}{k}\sum_{j=1}^k \nabla_\theta \log \pi_\theta(y_j,z_j|x)\,(r(x,y_j,y^*)-\bar r)\;-\; \frac{\tau}{2}\nabla_\theta\Big(\log \frac{\pi_\theta(y_j,z_j|x)}{\pi_{\theta_i}(y_j,z_j|x)}\Big)^2 ] (Eq. (3), Section 2.3.2)

What Eq. (3) is doing in plain language: - The first term is a policy-gradient-like update with a baseline (\(\bar r\)) to reduce variance: increase probability of rewarded samples and decrease probability of unrewarded samples relative to the batch mean (Section 2.3.2). - The second term is an explicit L2 penalty on the log-probability ratio, which further discourages large deviations from the reference policy (Eq. (3), Section 2.3.2). - A notable operational detail: because each iteration uses a different reference policy, they reset the optimizer at the start of each iteration (Section 2.3.2).

What’s missing (and cannot be inferred here): the report does not provide concrete optimizer hyperparameters for RL (e.g., Adam/SGD settings, learning rate, batch size, number of iterations, or \(\tau\) value) in the provided content. It also does not specify model architecture sizes (layers, hidden dimension, heads) for k1.5 in the excerpt.

3.4.3 Why no value function: encouraging exploration over classical credit assignment

  • The system explicitly excludes a value network (Section 2.3.2).
  • The report argues that classic value-based credit assignment can be misaligned with long-CoT training because a “wrong” intermediate step can still be useful if the model later recovers and reaches the correct answer; penalizing such exploration could reduce learning of trial-and-error behaviors (Section 2.3.2).
  • Mechanistically, they choose to rely on final-answer correctness as the main reward signal, which makes the training pressure: “any chain-of-thought that ends correct is good,” rather than “every step must look good under a value function” (Section 2.3.2).

3.4.4 Length control: shaping token efficiency with a length penalty

  • They observe “overthinking”: RL training drives responses to become much longer, often improving performance but increasing cost and reducing human preference (Section 2.3.3).
  • They introduce a length-based reward computed within the k samples for the same prompt (Section 2.3.3):
  • Let len(i) be the token length of sample \(i\), and define \(min\_len\) and \(max\_len\).
  • If all samples have equal length, length reward is 0.
  • Otherwise define \(\lambda = 0.5 - \frac{len(i)-min\_len}{max\_len-min\_len}\).
  • Then:
    • If the sample is correct (\(r=1\)), it receives \(len\_reward(i)=\lambda\) (rewarding shorter correct solutions relative to longer correct ones).
    • If the sample is incorrect (\(r=0\)), it receives \(len\_reward(i)=\min(0,\lambda)\), which explicitly penalizes long wrong answers (Section 2.3.3).
  • They state the length reward is added to the original reward with a weighting parameter (not specified in the excerpt) (Section 2.3.3).
  • They also “warm up” the length penalty: first train without it, then apply a constant penalty afterward, because it can slow early learning (Section 2.3.3).

3.4.5 Sampling strategy: selecting which problems to spend RL compute on

  • Two sampling methods are described (Section 2.3.4):
  • Curriculum sampling: start with easier tasks, gradually shift to harder tasks; rationale is that early models rarely solve very hard prompts, wasting compute (Section 2.3.4).
  • Prioritized sampling: maintain a success rate \(s_i\) per problem and sample proportional to \(1-s_i\), concentrating training on weak areas (Section 2.3.4).
  • The report later shows curriculum sampling improves performance vs uniform sampling on a mixed dataset (Figure 9; Section 3.5).

3.4.6 Reward/verification pipelines: making “correctness” reliable across domains

Because Eq. (1) optimizes correctness reward, evaluator quality is central.

(a) RL prompt set curation (to reduce reward hacking and improve evaluability)

  • They define a high-quality RL prompt set by: diverse coverage, balanced difficulty, and accurate evaluability (Section 2.1).
  • Difficulty labeling is model-based:
  • For each prompt, an SFT model samples 10 answers at “relatively high” temperature.
  • The pass rate is used as a difficulty proxy (lower pass rate = harder) (Section 2.1).
  • Anti–reward-hacking filters:
  • They remove prompt types prone to false-positive verification (multiple choice, true/false, proof-based) (Section 2.1).
  • They introduce an “easy-to-hack” detector: ask a model to guess answers without CoT; if it guesses correctly within \(N\) attempts, remove the prompt. They report using \(N=8\) (Section 2.1).

(b) Coding: automatic test case generation

  • Many web coding problems lack tests, so they build a pipeline using:
  • CYaRon as a test generation library,
  • the base k1.5 model to generate test cases from the problem statement + CYaRon usage statement (Section 2.3.5).
  • Filtering logic (precise thresholds given):
  • Generate 50 test cases.
  • Sample 10 ground-truth submissions per test case and execute them.
  • Keep a test case if at least 7/10 submissions match outputs.
  • Keep a problem if at least 9/10 submissions pass the full selected test set (Section 2.3.5).
  • Reported yield from a 1,000-problem sample:
  • 614 problems do not require a special judge.
  • 463 test case generators produce at least 40 valid tests.
  • 323 problems are included in the training set (Section 2.3.5).

(c) Math: chain-of-thought reward model for equivalence/format issues

  • They highlight a core issue: correct math answers can have different surface forms (Section 2.3.5).
  • They train two reward models, each with ~800k data points:
  • Classic RM: value-head scalar classifier taking (question, reference answer, response) (Section 2.3.5).
  • Chain-of-Thought RM: generates step-by-step reasoning then a JSON correctness judgment (Section 2.3.5).
  • Manual spot-check accuracies (as reported):
  • Classic RM: ~84.4
  • CoT RM: ~98.5
    and they use the CoT RM in RL training (Section 2.3.5).

(d) Vision RL data

  • Vision RL data is drawn from three categories (Section 2.3.5):
  • Real-world visual reasoning (science questions, location guessing, chart/data analysis).
  • Synthetic visual reasoning (procedural images/scenes targeting spatial/geometric/object reasoning).
  • Text-rendered data (render text/code/structured data as images to ensure consistent behavior across text vs screenshot-like inputs).

3.4.7 Infrastructure: making long-context RL feasible (partial rollouts + hybrid train/infer)

System/data pipeline diagram in words (explicit flow)

  1. A master coordinates rollout and training (Figure 3a; Section 2.6.1).
  2. Rollout workers query the current policy model to generate trajectories (model outputs that include long CoT and final answers) (Figure 3a; Section 2.6.1).
  3. Generated trajectories are sent for evaluation:
  4. Code-related outputs go through a code execution service / sandbox.
  5. Other domains use corresponding reward models/verifiers (Figure 3a shows separate reward models for Code/Math/K-12/Vision; Section 2.6.1).
  6. Trajectories (and their rewards) are stored in a replay buffer (Figure 3a; Section 2.6.1).
  7. Trainer workers sample from the replay buffer and compute gradients to update the policy model (Figure 3a; Section 2.6.1).
  8. Updated weights are pushed back to rollout workers for the next iteration (Figure 3a; Section 2.6.1).

Partial rollouts (key long-context technique)

  • Long rollouts are expensive because generating 128k tokens from scratch each time is slow.
  • Their partial rollout technique enforces a fixed output token budget per iteration (Section 2.6.2; Figure 3b):
  • If a response exceeds the token limit, the unfinished portion is saved to the replay buffer.
  • In the next iteration, generation continues from the saved prefix instead of regenerating from the start.
  • Only the current segment is on-policy; previous segments are reused (Section 2.6.2).
  • They also mention:
  • Asynchronous rollout workers can process short tasks while others continue long trajectories, improving utilization (Section 2.6.2).
  • Some segments can be excluded from loss computation “to further optimize” (the excerpt does not specify the exact rule) (Section 2.6.2).
  • Repeat detection can early-terminate repetitive generations and can apply penalties (Section 2.6.2).

Hybrid deployment (Megatron training + vLLM inference in one pod)

  • They alternate between:
  • Training phase: Megatron trains; then offloads GPU memory and prepares weights for vLLM (Section 2.6.3; Figure 4).
  • Inference/rollout phase: vLLM starts (dummy weights), receives updated weights via Mooncake over RDMA, runs rollouts, then is terminated to fully release GPU memory (Section 2.6.3; Figure 4).
  • Then Megatron reloads GPU memory and resumes training (Section 2.6.3).
  • Reported switching times:
  • “less than one minute” from training → inference, and “about ten seconds conversely” (Section 2.6.3).
  • They introduce a checkpoint-engine shim and use etcd as global metadata coordination (Section 2.6.3).

Code sandbox optimization

  • They implement a sandboxed code execution service (Section 2.6.4) and report performance comparisons (Table in Section 2.6.4):
  • Container startup time: Docker 0.12 s vs Sandbox 0.04 s.
  • Max container start rate (16-core machine): Docker 27 containers/sec vs Sandbox 120 containers/sec.

3.4.8 Long2short: compressing long-CoT into short-CoT

The report treats long-CoT as powerful but costly, and proposes “long2short” transfers (Section 2.4).

  • Model merging: average weights of a long-CoT and short-CoT model to get a new model without training (Section 2.4).
  • Shortest rejection sampling: sample each prompt \(n\) times (they use \(n=8\)) and SFT on the shortest correct response (Section 2.4).
  • DPO (Direct Preference Optimization): build preference pairs where:
  • positive = shortest correct solution,
  • negative = longer solutions (including wrong long ones and also correct long ones that are 1.5Ă— longer than the positive) (Section 2.4).
  • Long2short RL: run a second RL phase starting from a model chosen for performance-token tradeoff, apply the length penalty (Section 2.3.3), and reduce maximum rollout length to discourage long outputs (Section 2.4).

3.4.9 A worked micro-example (single input → output → reward → update)

This example is illustrative and mirrors the mechanisms described in Sections 2.3.1–2.3.3 (no hidden assumptions beyond what the report defines).

  1. Input: a math problem \(x\) with ground-truth answer \(y^*\).
  2. Sampling: at iteration \(i\), sample \(k\) completions \((y_j,z_j)\) from the reference policy \(\pi_{\theta_i}\) (Section 2.3.2).
  3. Scoring: for each sample, compute \(r_j=r(x,y_j,y^*)\in\{0,1\}\) using the math Chain-of-Thought RM/verifier (Section 2.3.5).
  4. Baseline: compute \(\bar r=\frac{1}{k}\sum_j r_j\) (Section 2.3.2).
  5. Optional length shaping: compute len_reward(j) using the within-prompt min/max lengths, rewarding shorter correct samples (Section 2.3.3), and add it (with some weight) to the reward used for optimization (Section 2.3.3).
  6. Update: apply Eq. (3):
  7. If a sample is correct (\(r_j=1\)) and \(\bar r<1\), the term \((r_j-\bar r)\) is positive, increasing \(\log \pi_\theta(y_j,z_j|x)\), i.e., making that (answer + reasoning trace) more likely.
  8. If a sample is incorrect (\(r_j=0\)) and \(\bar r>0\), the term is negative, decreasing its likelihood.
  9. The squared log-ratio penalty prevents \(\pi_\theta\) from moving too far from \(\pi_{\theta_i}\) in one step (Eq. (3), Section 2.3.2).

4. Key Insights and Innovations

  • (1) Long-context RL scaling to 128k with partial rollouts
  • Novelty in this report is not “long context” alone, but the combination of (i) training RL with very long rollouts and (ii) making it computationally feasible via partial rollouts that reuse prior trajectory segments (Introduction; Section 2.6.2; Figure 3b).

  • Significance: they argue context length acts like a “search step” budget, enabling planning/reflection/correction behaviors without explicit tree search (Introduction; Section 2.3.1).

  • (2) A KL-regularized, off-policy mirror-descent-style policy optimization for long-CoT

  • They formulate RL with long-CoT and use an online mirror-descent variant with a KL term to stabilize updates (Section 2.3.2; Eq. (2)).
  • The presented gradient (Eq. (3)) combines a REINFORCE-like baseline term with an explicit squared log-ratio penalty, and uses samples from the reference policy (off-policy relative to \(\pi_\theta\)) (Section 2.3.2).

  • Significance: in ablations, they argue “negative gradients” (penalizing wrong responses) improve sample complexity vs ReST, which does not penalize incorrect samples (Section 3.5; Figure 10).

  • (3) Practical reward/verification engineering (especially math CoT reward modeling)

  • They report that a CoT-augmented reward model improves manual spot-check accuracy from ~84.4 to ~98.5 and is used to provide more reliable RL feedback (Section 2.3.5).

  • Significance: higher evaluator accuracy reduces reward noise and makes final-answer reward more trustworthy for RL.

  • (4) A “token efficiency” toolkit: length penalty + long2short transfer

  • They directly address RL-driven overlong reasoning via a length-based reward shaping and a warmup schedule (Section 2.3.3).

  • They then propose long2short transfer methods (merging / shortest-RS / DPO / long2short RL) and show long2short RL yields the best token-efficiency tradeoff in their comparisons (Section 3.4; Figure 7).

  • (5) Systems contribution: hybrid training/inference deployment + fast sandboxing

  • The hybrid Megatron↔vLLM switching framework and the code sandbox throughput improvements are concrete engineering contributions enabling high-throughput RL with verifiers (Section 2.6.3–2.6.4; Figure 4; sandbox table).

5. Experimental Analysis

Evaluation methodology (datasets, metrics, setup)

  • Benchmarks are grouped into (Section 3.1; Appendix C):
  • Text: MMLU (EM), IF-Eval (Prompt Strict), CLUEWSC (EM), C-Eval (EM).
  • Reasoning: HumanEval-Mul (Pass@1), LiveCodeBench (Pass@1), Codeforces (Percentile), AIME 2024 (Pass@1), MATH-500 (EM).
  • Vision: MMMU (Pass@1), MathVision (Pass@1), MathVista (Pass@1).
  • The report distinguishes two evaluated variants:
  • Long-CoT model (Tables/Figures for long-CoT: Figure 1; Table 2).
  • Short-CoT model (Figure 2; Table 3), produced using SFT + RL + long2short methods (Section 3.2; Section 2.4).

Main quantitative results (with specific numbers)

Long-CoT results (Table 2; Figure 1): - Math: - MATH-500 (EM): 96.2 - AIME 2024 (Pass@1): 77.5 - Code: - Codeforces (Percentile): 94 - LiveCodeBench v5 24.12–25.2 (Pass@1): 62.5 - Vision: - MathVista-Test (Pass@1): 74.9 - MMMU-Val (Pass@1): 70.0 - MathVision-Full (Pass@1): 38.6

Short-CoT results (Table 3; Figure 2): - Text: - MMLU (EM): 87.4 - IF-Eval (Prompt Strict): 87.2 - CLUEWSC (EM): 91.7 - C-Eval (EM): 88.3 - Reasoning: - MATH-500 (EM): 94.6 - AIME 2024 (Pass@1): 60.8 - HumanEval-Mul (Pass@1): 81.5 - LiveCodeBench v4 24.08–24.11 (Pass@1): 47.3 - Vision: - MathVista-Test (Pass@1): 70.1 - MMMU-Val (Pass@1): 68.0 - MathVision-Full (Pass@1): 31.0

Analyses supporting core claims

  • Long context scaling correlates with performance and longer outputs
  • Figure 5 shows, for an internal smaller long-CoT model trained on math prompts, both accuracy and response length increase across iterations, with harder benchmarks showing steeper length increases (Section 3.3; Figure 5 caption notes scores are from a smaller internal model).

  • Figure 6 plots performance vs mean token length and reports positive trends (slopes shown on plots), supporting the report’s narrative that longer reasoning traces are associated with better problem-solving (Section 3.3; Figure 6).

  • Long2short token efficiency comparison

  • Figure 7 compares token length vs accuracy for MATH500 and AIME2024 across methods (DPO, merge, merge+RS, long2short RL, etc.).

  • A concrete number reported in text (Section 3.4):
    > k1.5-short w/ rl achieves 60.8 Pass@1 on AIME2024 (averaged over 8 runs) using 3,272 tokens on average.
    (Section 3.4; Figure 7 is the referenced plot.)

  • Ablations

  • Model size vs context length: Figure 8 shows that a smaller model can reach comparable performance by using longer RL-optimized CoT, while larger models are generally more token-efficient; the report interprets this as context scaling being a viable alternative lever under test-time compute constraints (Section 3.5; Figure 8).
  • Negative gradients vs ReST: Figure 10 shows their method outperforms ReST (which “fits best responses” without penalizing incorrect ones) in sample complexity on multiple benchmarks (Section 3.5; Figure 10).
  • Curriculum sampling: Figure 9 shows improved performance over a baseline uniform sampler when transitioning to hard questions after warmup (Section 3.5; Figure 9).

How convincing are the experiments (based on provided content)?

  • Strengths:
  • The report provides multi-domain benchmark coverage (text, math, code, vision) (Section 3.1; Tables 2–3).
  • It includes targeted ablations tied to core design choices: context length scaling, negative gradients, curriculum sampling (Section 3.5; Figures 8–10).

  • It discusses evaluator quality for math reward modeling with explicit accuracy numbers (~84.4 vs ~98.5) (Section 2.3.5), which is crucial for RL credibility.

  • Gaps / missing details (important for full reproducibility, not provided here):

  • No model architecture specs for k1.5 (layers/width/heads), no RL optimizer settings (learning rate, batch size, \(\tau\)), no training compute budget (tokens or PF-days), and no hardware configuration are included in the provided excerpt (contrast with the prompt requirement; these details are simply absent).
  • Some benchmark scores are noted as coming from an intermediate model due to “version shift” for IF-Eval (Appendix C.1), which complicates strict comparability.

6. Limitations and Trade-offs

  • Reliance on verifiability / evaluator correctness
  • The RL framework assumes rewards can be reliably computed (Section 2.3.1).
  • They explicitly remove multiple-choice, true/false, and proof-based prompts to avoid false-positive verification and reward hacking (Section 2.1), which narrows the task distribution.

  • Even for math, they need a high-accuracy reward model; they report improved spot-check accuracy with CoT RM (Section 2.3.5), but reward modeling remains a potential failure point.

  • Overthinking vs performance

  • RL increases response length (“overthinking”), improving performance but raising training and inference cost and possibly harming user preference (Section 2.3.3).

  • Length penalty helps but may slow early training, requiring warmup schedules (Section 2.3.3).

  • Compute and systems complexity (despite “simplistic RL” algorithmically)

  • While the RL algorithm avoids value functions and tree search (Introduction; Section 2.3.2), the infrastructure is non-trivial:
    • partial rollouts, replay buffers, hybrid deployment, RDMA weight transfer, sandboxing (Section 2.6).

  • The report does not quantify end-to-end compute cost for RL training in the provided content, making it hard to judge practical accessibility.

  • Potential domain skew

  • The prompt set is curated for easy verification and anti-hacking; this may bias training toward domains where correctness is cleanly checkable (math with RM, code with tests, certain vision QA forms) (Sections 2.1 and 2.3.5).

  • Incomplete transparency in the provided excerpt

  • Pretraining and data pipelines are described qualitatively with some numeric dataset sizes for vanilla SFT (Section 2.5.2), but exact pretraining scale (total tokens, steps) and core model hyperparameters are not included here.

7. Implications and Future Directions

  • How this changes the landscape (as supported by this report)
  • The report argues that context length is a key scaling dimension for RL with LLMs, because longer contexts let the model carry richer “search histories” and effectively increase the number of reasoning steps without explicit tree search (Introduction; Section 2.3.1; Section 3.3; Figures 5–6).

  • It suggests that strong reasoning performance can be reached with a relatively straightforward RL loop if you solve evaluator quality and long-context infrastructure (Introduction; Sections 2.3 and 2.6).

  • Follow-up research directions explicitly suggested or implied

  • Improving credit assignment while preserving exploration (they note this as future intrigue) (Conclusions).
  • Reducing overthinking without hurting exploration (Conclusions; Section 2.3.3).

  • Improving efficiency and scalability of long-context RL training (Conclusions).

  • Practical applications / downstream use cases

  • Multimodal reasoning that requires integrating text + images (MathVista/MMMU/MathVision benchmarks; Section 3.1; Tables 2–3).

  • Code generation and competitive programming-style problem solving, where correctness is verifiable via execution (Codeforces, LiveCodeBench; Section 3.1; Tables 2–3; Section 2.3.5; Section 2.6.4).

  • Repro/Integration Guidance (grounded in this report’s design choices)

  • Prefer this approach when:
    • You have verifiable tasks (unit tests, deterministic checkers, or high-accuracy reward models) to supply reliable rewards (Section 2.3.1; Section 2.3.5).
    • You can benefit from long-context reasoning and can afford infrastructure to handle long rollouts (Sections 2.6.2–2.6.3).
  • Use long2short when:
    • You want long-CoT performance but have strict inference token budgets; the report’s long2short RL and related methods target this (Section 2.4; Figure 7).
  • Be cautious when:
    • Tasks are hard to verify (risk of reward hacking / false positives); the report itself curates away several such prompt types and uses an explicit “easy-to-hack” filter with \(N=8\) attempts (Section 2.1).
    • You cannot support the required rollout/training orchestration; the system relies on partial rollouts, hybrid deployment, and fast sandboxing for throughput (Section 2.6).