ROBO SWE Benchmark

Leaderboard

Share of tasks resolved, averaged over each setup's counted passes. A setup is a model plus its coding harness and reasoning setting — the whole system is what's measured.

● Updated

Intervals are 95%: a 2,000-resample task bootstrap for multi-pass setups, Wilson for single-pass binary scores. Overlapping intervals mean the ordering between those setups is not settled. Different harnesses expose different tools, so compare setups, not bare model names.

Results by domain

Solve rate (%) for each completed setup in each part of the robotics stack. The outlined cell is the best result in that column.

Where the tasks come from

Task counts per domain, from the same comparison set used for the leaderboard.

Does more reasoning help?

A paired comparison: the same model and harness at two reasoning-effort settings, on the same tasks.

What a task looks like

Every task starts with a real issue and a merged fix. The agent sees the issue and the repository before the fix; the fix and the developer's tests stay hidden.

FROM THE BENCHMARK · ROS 2ros2 / rosbag2 · #1812 ↗

“ros2 bag convert ignoring messages with unknown types”

The agent receives

The original issue and the repository at the commit before the fix, inside its historical environment.

issue + source tree

The evaluator keeps hidden

The developer's fix and tests. One check captures the bug; seven cover behavior that already worked.

1 bug check · 7 regression checks

Before the fixThe bug check fails

Agent writes a patchAny correct implementation

After a correct fixAll required checks pass

The reference change is 11 lines across two files — small, but it still requires understanding the issue and the system around it.

How it's built and scored

Construction checks that each issue, fix and test suite can support a reproducible evaluation before it becomes a task.

01

Find the issue

Connect the reported behavior to a merged fix and the tests that exercise it.

02

Rebuild the context

Prepare the repository before the fix, with its dependencies and test command.

03

Check both states

Confirm the bug check fails before the fix and passes after it, while checking regressions.

04

Package the task

Keep the reference solution and evaluator hidden, and record the task's provenance.

What counts as a passThe tests that separate the broken and fixed states must pass with the agent's patch, and every required test that passed before must still pass.
One-hour budget per attemptAn agent that uses the full hour without solving the task has a valid failed attempt. Infrastructure failures — a failed login, a provider outage — are not evidence about the model and receive no grade.
Repeated passesThe main comparison runs each setup three times. Some setups have a single complete pass; those are labelled and use Wilson intervals.
Reproducibility exclusionsHistorical software can depend on packages that later disappear. A task whose pinned environment cannot be rebuilt is excluded for every setup alike, independent of which models solved it.

What this does not measure

Useful evidence, with clear boundaries.

Tests are a measurement, not the whole specification.

Developer tests may omit important behavior, or require details the issue doesn't fully state.

Human review is ongoing.

Automated before-and-after checks, AI review notes and human judgment are different evidence. The benchmark is not described as fully human-verified.

Public code may have appeared in training.

The repositories and their fixes are public. We can't rule out contamination, or turn familiarity with a repository into a claim about general capability.

A container is not a physical robot.

Passing software tests does not establish real-world robustness, deployment readiness or robot safety.

Changelog

The leaderboard is updated as evaluations finish. Each change is recorded here.

9 Nov 2026Public release. Eight setups across Claude Code, Codex and OpenCode; six complete, two in progress. One task excluded for every setup because its pinned environment could not be rebuilt.

Paper and citation: TBD.

ROBO SWE: can coding agents repair the software robots run on?