Benchmarks: public evaluations, built the way we build for labs.

Real tasks, hidden verifiers, repeated passes and honest intervals. Each benchmark keeps a live leaderboard here, updated as new evaluations finish.

Live · Robotics software engineering

Can coding agents repair the software robots run on? 236 real issues from 28 robotics repositories, scored with the developers' own hidden tests.

ROS 2Simulation & physicsRobotics algorithmsRobot learning

More benchmarks are in development. Get in touch if you'd like early access.

Need an eval that doesn't exist yet?

We design private benchmarks and held-out evals for model builders, in the domains your models are weakest.