Encyclopedia

Reasoning model

Large language model designed for reasoning tasks

Wikipedia contributorsAdapted for Fathomly · View authors

8 min read
Source updated September 5, 2026

Reasoning language models (RLMs) or large reasoning models (LRMs) are large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations.

01History

2024

In September 2024, OpenAI released o1-preview, an LLM with enhanced reasoning. The full version, o1, followed in December 2024. OpenAI also began sharing results with its successor, o3.

The development of reasoning LLMs has illustrated what Rich Sutton called the "bitter lesson": that scaling computing power often outperforms methods that rely on specific human insights. For example, the Generative AI Research Lab (GAIR) explored complex methods, such as tree search and reinforcement learning, to replicate o1's capabilities. In their "o1 Replication Journey" papers they reported that knowledge distillation (training a smaller model to imitate o1's outputs) worked surprisingly well. This highlighted the effectiveness of distillation in this context.

Alibaba released reasoning versions of its Qwen LLMs in November 2024. In December 2024, the team introduced QvQ-72B-Preview, an experimental visual reasoning model.

In December 2024, Google introduced Deep Research in Gemini, a feature that runs multi-step research tasks.

On December 16, 2024, an experiment with a Llama 3B model showed that by scaling test-time computing, a relatively small model could outperform a much larger Llama 70B model on challenging reasoning tasks. This suggested that better inference strategies can unlock useful reasoning capabilities even in small models.

2025

In January 2025, DeepSeek released R1, a model with performance comparable to o1, at a lower cost. The release demonstrated the effectiveness of Group Relative Policy Optimization (GRPO). On January 25, 2025, DeepSeek added a feature to DeepSeek R1 that lets the model search the web while it reasons, making it easier to combine retrieval with reasoning. The effectiveness of distillation for reasoning models was shown in works such as s1-32B, which achieved strong performance through budget forcing and scaling methods.

On February 2, 2025, OpenAI released Deep Research based on their o3 model, allowing users to initiate complex research tasks and generate comprehensive reports that incorporate various sources from the web.

OpenAI called GPT-4.5 its "last non-chain-of-thought model", and implemented with GPT-5 a router that selects a model based on the difficulty of the task.

2026

In January 2026, Moonshot AI released Kimi K2.5, an open-source, 1-trillion-parameter mixture of experts (MoE) model with 32 billion active parameters. That was followed by the release of Kimi K2.6 in April 2026, where both Kimi models use an “Agent Swarm” system that dynamically decomposes tasks into sub-agents for reasoning and execution, enabling more scalable multi-step problem-solving than a with single sequential reasoning chain. However, both Kimi models, despite being identical with regard to parameters, have differences in their uses of Agent Swarm, with K2.5 utilizing a 100-sub-agent format and K2.6 utilizing a 300-sub-agent one, which enhances the latter's ability for task coordination.

02Supervised finetuning

A large language model (LLM) can be fine-tuned on a dataset of reasoning tasks paired with example solutions and step-by-step (reasoning) traces. The fine-tuned model can then produce its own reasoning traces for new problems.

Because human-written traces are costly to collect, researchers have proposed ways to build such datasets automatically. In rejection sampling finetuning (RFT), new reasoning traces are gathered in a loop:

Sample a task prompt.
Generate many reasoning traces for the prompt.
Use a verifier to remove reasoning traces with a wrong final answer, and optionally remove duplicates.

03Reinforcement learning

A pretrained language model can be further trained with reinforcement learning (RL). In the RL formalism, a generative language model is a policy $\pi$ . A task prompt is an environmental state $x$ , and the model's response is an action $y$ . The probability that, given $x$ , the model responds with $y$ , is $\pi (y|x)$ .

Training a reasoning language model with RL means constructing a reward model $r(x,y)$ to guide the RL process. Intuitively, the reward says how good a response is for a prompt. For a reasoning task, the reward is high if the response solves the task and low if it does not.

A response $y$ may be broken down into multiple steps, written $y_{1},y_{2},\dots ,y_{n}$ .

Most recent systems use policy-gradient methods such as Proximal Policy Optimization (PPO) because PPO constrains each policy update with a clipped objective, which stabilises training for very large policies.

Outcome reward model

An outcome reward model, or outcome-supervised RM (ORM), gives the reward for a step $r(x,y_{1},\dots ,y_{i})$ based on the final answer: $r(x,y_{1},\dots ,y_{i})=r(x,y_{n})$ . Such models are often called "verifiers".

For tasks with answers that are easy to verify, such as math word problems, the outcome reward can be binary: 1 if the final answer is correct, 0 otherwise. If automatic verification is hard, humans can label answers as correct or incorrect, and those labels can be used to fine-tune a base model that predicts the human label. For tasks such as creative writing, where quality is not simply true or false, one can train a reward model on human ranked preference data, as in reinforcement learning from human feedback. A base model can also be fine-tuned to predict, from a partial thinking trace $x,y_{1},\dots ,y_{m}$ , whether the final answer will be correct, and this prediction can serve as a binary reward.

The ORM is usually trained with logistic regression, i.e. by minimizing cross-entropy loss.

Given a process reward model (PRM} (see following), an ORM can be constructed by multiplying the total process reward during the reasoning trace by taking the minimum or by other ways of aggregating process rewards. DeepSeek used a simple ORM to train the R1 model.

Process reward model

A process reward model, or process-supervised RM (PRM), gives the reward for a step $r(x,y_{1},\dots ,y_{i})$ based only on the steps so far: $(x,y_{1},\dots ,y_{i})$ .

Given a partial thinking trace $x,y_{1},\dots ,y_{m}$ , a human can judge whether the steps so far are correct, without looking at the final answer. This yields a binary reward. Because human labels are costly, a base model can be fine-tuned to predict them. The PRM is usually trained with logistic regression on the human labels, i.e. by minimizing the cross-entropy loss between true and predicted labels.

As an example, a 2023 OpenAI paper collected 800,000 process labels for 75,000 thinking traces. A labeler saw a trace and marked each step as "positive" if it moved toward a solution, "neutral" if it was not wrong but did not help, and "negative" if it was a mistake. After the first "negative" label, the labeler stopped working on that trace and moved to another. The authors argued that labeling up to the first error was enough to train a capable PRM, even though labeling later steps could give richer signals.

To avoid human labels, researchers have proposed methods to create PRM without human labels on the processes. Inspired by Monte Carlo tree search (MCTS), the Math-Shepherd method samples multiple continuations until the end, starting at each reasoning step $y_{i}$ , and set the reward at that step to be either ${\frac {\#{\text{(correct answers)}}}{\#{\text{(total answers)}}}}$ in the case of "soft estimation", or ${\begin{cases}1&{\text{if one of the answers is correct}}\\0&{\text{else}}\end{cases}}$ in the case of "hard estimation". This creates process rewards from an outcome reward model (ORM), which is often easier or cheaper to construct. A PRM can then be trained on these labels. Some work has tried a fully MCTS approach.

One can also use an ORM to implicitly construct a PRM, similar to direct preference optimization.

Guided sampling

A trained ORM can be used to pick the best response. The policy generates several responses, and the ORM selects the best one. This implements a simple form of test-time compute scaling ("best-of-N").

A trained PRM can guide reasoning by a greedy tree search: the policy proposes several next steps, the PRM picks one, and the process repeats. This mirrors using an ORM to pick a whole response. Beam search performs better than greedy search.

"Lookahead search" is another tree search method. The policy proposes several next steps, then makes a short rollout for each. If a solution is found during rollout, the search stops early. Otherwise, the PRM scores each rollout, and the step with the highest score is chosen.

"Self-consistency" can be combined with an ORM. The model generates multiple answers, and the answers are clustered so that each cluster has the same final answer. The ORM scores each answer, scores in each cluster are summed, and the answer from the highest-scoring cluster is returned.

04Benchmarks

Reasoning models generally score higher than non-reasoning models on many benchmarks, especially on tasks requiring multi-step reasoning.

Some benchmarks exclude reasoning models because those models' responses take longer and cost more.

Humanity's Last Exam

The HLE benchmark tests expert-level reasoning across mathematics, humanities, and the natural sciences, and shows large performance gaps between models. State-of-the-art reasoning models score low on HLE, leaving room to them to improve. For example, the full reasoning model o3 reached 26.6%, while the lighter o3-mini-high (on text-only questions) reached 13%.

AIME

On the American Invitational Mathematics Examination (AIME), a difficult math competition, non-reasoning models usually solve under 30% of problems. Models that use reasoning methods score between 50% and 80%. While OpenAI's o1 maintained or slightly improved its accuracy from 2024 to 2025 on the AIME, o3-mini (high) reached a higher accuracy (80%) at a much lower cost (about 12 times cheaper).

o3-mini performance

According to OpenAI's January 2025 report on o3-mini, adjusting "reasoning effort" significantly affects performance, especially for STEM tasks. Moving from low to high reasoning effort raises accuracy on AIME 2024, GPQA Diamond, and Codeforces, typically by 10-30%. With high effort, o3-mini (high) achieved 87.3% on AIME (different from the MathArena AIME benchmark), 79.7% on GPQA Diamond, 2130 Elo on Codeforces, and 49.3 on SWE-bench Verified.

05Drawbacks

Computational cost

Reasoning models often need far more computing power while answering than non-reasoning models. On AIME, they were 10 to 74 times more expensive than non-reasoning counterparts. These models are also vulnerable to overthinking attacks that can intentionally slow down the reasoning process, causing increasing operational costs or even denial-of-service.

Generation time

Due to the tendency of reasoning language models to produce verbose outputs, the time it takes to generate an output increases greatly when compared to a standard large language model.

06Models

OpenAI

Gemini

DeepSeek

R1 (based on V3)
R1-Lite-Preview (test version based on V2.5)

Qwen

QvQ-72B-Preview, an experimental visual reasoning model launched on December 24, 2024, which integrates image understanding with verbal chain-of-thought reasoning.
QwQ-32B-Preview, an experimental text-based reasoning model released in late November 2024 that emphasizes complex, step-by-step analysis.

Anthropic

Claude Sonnet 3.7 has an adjustable amount of 'thinking' tokens.

Mistral AI

Magistral (medium & small)

xAI

Hugging Face

OlympicCoder-7B & 32B, as part of reproducing the R1 training openly (Open R1 project).

Watch videos about Reasoning modelExplainers and documentaries on YouTube (opens in a new tab)

Sources and credits

This article is adapted from the Wikipedia article “Reasoning model”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.

Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.