DHH says Sol trails Opus by 5 points at a 13th of cost

The Ruby on Rails team published Stage 3 of its Agents on Rails benchmark, seven new multi-step tasks each spanning three to eight tickets, from building Rails apps from scratch to working inside existing and at times messy legacy codebases. The point was to see how agents handle going back and building on their own work.

OpenAI's GPT-6.1 Sol came out as best overall on the Rails team's own read, with 60.8% success at $0.80 and 23 minutes 35 seconds per run. Anthropic's Opus 5.5 at max effort was the most accurate at 65.3%, and the Rails post notes it is also the most expensive and among the slowest. DHH put a number on that gap, $0.80 against $10.53 per run, and called the Sol result incredible performance from OpenAI.

DHH

@dhh

X

Sol completes the tasks less than 5 percentage points off Opus, but at less than a tenth of the cost!! $0.80 vs $10.53.

What else placed

GPT-6 Astra at medium effort was fastest at 16 minutes 36 seconds with 54% success, and its max effort setting managed 22 minutes 36 seconds at 62.3%, which is within a point of Sol on accuracy and faster. The cheapest run was GPT-6 Luna at $0.19 with 43.5% success. The Rails team also rolled every task from every stage into one central leaderboard, and the full Stage 3 report was written up by Evil Martians.

The business read

This is a benchmark on Rails tickets, not your codebase, and it is one team's methodology rather than an independent audit. But the shape of the result is the useful bit for anyone running agents all day. The accuracy leader costs roughly thirteen times the price of the balanced pick for under five percentage points of success rate, and it is slower on top of that. If you are a solo founder paying per run and shipping dozens of small tickets a week, that is the difference between an agent bill you notice and one you do not.

The honest counter is that a five point success gap compounds when a failed run means you re-run it, re-prompt it, or fix it by hand, and none of those costs show up in a per-run price. Nobody in these posts has priced that out. What the numbers do support is that defaulting every task to the most accurate model is now a choice with a visible cost attached, and the spread between the cheapest and the most accurate option in this run was $0.19 to $10.53.