A robot is also a software stack.
A robot's behavior depends on much more than its policy. Messages must arrive, coordinate frames must agree, controllers must respect their limits, and simulators must compute the right physics.
Repairing that software is a distinct engineering problem. A plausible patch can compile and still mishandle a message, change a controller's behavior, or break another part of a simulation. We want to measure whether coding agents can resolve those issues in the codebases where they actually occur.
ROBO SWE is a benchmark of real issues and merged fixes from robotics repositories. An agent receives the issue and the repository before the fix. It works in a prepared environment, changes the code, and submits its patch. Hidden tests written by the original developers evaluate the result.
Can an agent understand the issue, find the relevant behavior, and repair it without breaking what already works?
The benchmark spans ROS 2, simulation and physics, robotics algorithms, and robot learning. These are software tasks in containers — not physical robot trials.
Keep a bag conversion working.
A rosbag is a recording of robot messages. In one task, converting a recording encounters a message type whose supporting package is unavailable. The issue asks for conversion to keep processing those messages.
“ros2 bag convert ignoring messages with unknown types”
The original issue and the repository at the commit before the fix, inside its historical environment.
issue + source treeThe developer's fix and tests. One check captures the bug; seven cover behavior that already worked.
1 bug check · 7 regression checksThe reference change spans 11 changed lines in two files. A small patch still requires understanding the issue and the surrounding system.
This example makes the evaluation rule concrete. Passing the new bug check is necessary, but it is not enough: the patch must also preserve the behavior covered by the existing checks. The hidden reference patch is evidence that the task can be solved, rather than a solution the agent can read.
From a real issue to a repeatable task.
ROBO SWE's tasks start with an issue, a pull request that fixes it, and developer tests. Construction checks whether those pieces can support a reproducible evaluation.
Find the issue
Connect the reported behavior to a merged fix and the tests that exercise it.
Rebuild the context
Prepare the repository before the fix, with its dependencies and test command.
Check both states
Confirm the bug check fails before the fix and passes after it.
Package the task
Keep the reference solution and evaluator hidden; record provenance.
The agent-facing instruction uses the original issue text. The task files pin the repository and environment; evaluation keeps the developer's tests separate from the agent's workspace.
What a passing result means
The tests that distinguish the broken and fixed states must pass with the agent's patch, and required tests that passed before must continue to pass. A one-hour agent budget bounds each attempt. A model that spends that full budget without solving the task has a valid failed attempt; a failed login or provider interruption is not evidence of model ability and receives no model grade.
Why environment provenance matters
Historical software can depend on packages that later disappear. The current comparison excludes one such task uniformly for all setups after its exact pinned dependency could not be recovered. Exclusions are determined by reproducibility, independent of which models solved the task.
Four domains, one evaluation question.
Measure the whole attempt, and show the uncertainty.
The unit of comparison is a setup: a model, its coding harness, and its reasoning setting. The primary comparison repeats each setup three times. Additional models may have one complete pass or evaluations still in progress.
A setup's solve rate is the mean of its task scores over its counted passes. Repeating an attempt helps distinguish consistent performance from a favorable single run. Uncertainty intervals describe variation across the sampled tasks, rather than promising the same result on every new codebase.
Resolve rate on real robotics issues
Snapshot · 9 Nov 2026| # | Setup | Solve rate |
|---|---|---|
| 01 | Claude Opus 5.5Claude Code · medium | 56.5% |
| 02 | GPT-6.1 SolCodex · max | 54.1% |
| 03 | GPT-6.1 SolCodex · medium | 52.5% |
| 04 | GPT-6 AstraCodex · medium | 51.4% |
| 05 | GLM-5.3OpenCode · default | 46.2% |
| 06 | Claude Sonnet 5.5Claude Code · medium | 45.8% |
| — | Claude Fable 5.1In progressClaude Code · medium · 136/236 tasks | 52.9% |
| — | Kimi K3.1 ProIn progressOpenCode · default | 43.9% |
Read these numbers as measurements of the complete setup. Different harnesses can expose different tools and interaction patterns. An overlapping interval is a reason to be cautious about a strict ordering — not a basis for declaring one model universally better.
One success is different from three successes.
A single run can hide instability. An agent may find the relevant code quickly on one attempt and spend another following an unhelpful path. We therefore inspect both average solve rate and the outcomes of repeated runs.
Solved in every run measures consistency. Solved in any run shows whether a setup found a working patch at least once. Neither replaces the average, and a three-run pattern does not by itself prove what a model knows.
Does thinking longer change the outcome?
ROBO SWE also compares Sol at medium and max reasoning effort. The useful question is a paired one: on the same tasks, how much does the outcome change?
The interval includes zero. This snapshot does not establish a clear improvement from increasing reasoning effort.
Reasoning effort is part of the setup, rather than a promise of better performance. A small observed difference whose interval includes zero should be reported as uncertain. Time and token cost matter too.
Useful evidence. Clear boundaries.
Tests are a measurement, not the whole specification.
Developer tests may omit important behavior or require details the issue does not fully state. Review of the issue, fix and evaluator remains important.
Human review is ongoing.
Automated before-and-after checks, AI review notes and human judgments provide different evidence. The benchmark should not be described as fully human verified.
Public code may have appeared in training.
The repositories and their historical fixes are public. We cannot rule out contamination or turn familiarity with a repository into a claim about general capability.
A container is not a physical robot.
Passing software tests does not establish real-world robustness, deployment readiness, or robot safety. Those require further evaluation.
Start with the issue. Follow the result.
See the current model comparisons, results by domain, and how the benchmark is evaluated — updated as new evaluations finish.
This post is a dated snapshot of ROBO SWE as of 9 November 2026. All numerical figures come from the same comparison set. Background reading: the rosbag2 example · Harbor evaluation framework.