The Agents on Rails Files: enter lemans, a Ruby harness for fair agent races

Why build our own harness?

We’ve briefly told the origin story before: the benchmark started on Harbor, and we wanted a harness of our own, in Ruby, that we could bend to every new task. With encouragement from the Rails Foundation, that became lemans. The principles we started with still hold:

• Minimal boilerplate. A task is an instruction with a few lines of metadata, a patch that prepares the app (plants a bug or removes a feature), a golden patch that solves it, and a Ruby test. This same anatomy has carried us from atomic Stage 1 tasks to multi-step Stage 3 tasks.
• CLI-first. lemans is still driven from the command line, so engineers and agents use it the same way. For humans, there’s also a web viewer with long agent transcripts (trajectories, in eval speak) that are much more pleasant to read in a browser.
• Fair grading. The first post covers how we design the checks. On the harness side, the sandbox went through several rounds of hardening, most prompted by real escape attempts.
• Ruby all the way. lemans has a flexible plugin system in the language we all know and love. This means a custom agent or sandboxing backend is just a Ruby file it loads on the fly. And because conventional graders are plain Ruby tests, anyone can read and tinker with them, too.

Getting started with lemans

Let’s take a quick tour in case you want to repeat the benchmark on your own Rails app. lemans is a gem, and lemans init scaffolds a new bench with two example tasks:

gem install lemans
lemans init my-bench

my-bench/
├── README.md
├── bench.yml                     # the configuration: environment, agent, verification
├── environment/Dockerfile        # the shared sandbox image
└── tasks/
    ├── hello-world/
    │   ├── instruction.md        # the task instructions (with frontmatter)
    │   ├── environment.patch     # prepares the app: plants a bug or removes a feature
    │   ├── solution.patch        # a known-good solution
    │   └── verification_test.rb  # the hidden grader
    └── example-task/             # a multi-step task: solution.1.patch, verification_test.1.rb, ...

Before benchmarking any model, prove the task itself. The oracle agent applies the reference solution, and grading must score 1.0. The nop agent changes nothing, and the task must score 0. A task that fails either check is broken, so any model score on it is noise.

cd my-bench
lemans run --backend docker --agent oracle
lemans run --backend docker --agent nop

Then race the models. lemans talks to them through RubyLLM, so any provider it supports works (including OpenRouter). Sandboxes come from Daytona by default, the same as in our benchmark, but local Docker is supported too:

export OPENROUTER_API_KEY=<your key>

lemans run \
  --backend docker \
  --model openrouter/openai/gpt-5.6-luna \
  --model openrouter/deepseek/deepseek-v4.1-flash \
  --runs-dir ./my-runs \
  --task hello-world \
  --attempts 3

Here are some useful options:

• --model overrides the model from bench.yml, written as <provider>/<model>, and a #effort suffix pins the reasoning effort (openrouter/openai/gpt-5.6-luna#xhigh).
• --runs-dir changes where the results go, ./runs by default.
• --task and --tag pick a subset of tasks, otherwise lemans runs them all in the bench.
• --attempts runs every task several times, so one lucky run doesn’t decide the score. The default is a single attempt.

Each trial lands in its own folder, with the transcript saved in Harbor’s ATIF format:

my-runs/
└── gpt-5.6-luna/
    └── hello-world__5Ko47JL/
        ├── result.json        # the trial record: score, usage, timings
        ├── agent.patch        # everything the agent changed
        ├── agent.result.json  # the agent's own result: messages, tokens, cost (miniswen)
        ├── trajectory.json    # the full transcript
        ├── verifier.log       # the grader's output
        └── checks.json        # the result of every check

lemans report prints the results in a readable format:

$ lemans report --runs-dir ./my-runs --aggregate model-task

model                task         score  time    cost     steps  tokens
deepseek-v4.1-flash  hello-world  3/3    2m 28s  $0.0011  8.3    26.7K
gpt-5.6-luna         hello-world  3/3    2m 19s  $0.0021  5      12.3K
deepseek-v4.1-flash  3 trials: 3 scored, 0 invalid, 3 solved (100%) · $0.0033 · pass@3 1/1 tasks (100%)
gpt-5.6-luna         3 trials: 3 scored, 0 invalid, 3 solved (100%) · $0.0062 · pass@3 1/1 tasks (100%)
total                6 trials: 6 scored, 0 invalid, 6 solved (100%) · $0.0095 · pass@3 2/2 tasks (100%)

While nobody is supposed to read what agents write anymore (especially if you’re prone to polite prompt injection), we still have to, and lemans-viewer makes this bearable:

lemans-viewer --runs-dir ./my-runs   # http://127.0.0.1:8089

Now, let’s see how a run gets its score.

The grader is a Ruby test

A task’s grader is usually plain Minitest, though it can be an executable verify script in any language. For a Rails app, it’s a regular Rails test that runs inside the app:

require "test_helper"

LemansReport.base_credit = 0.10

class VerifierTest < ActionDispatch::IntegrationTest
  include LemansReport::Assertions

  setup do
    @leaf = leaves(:welcome_page)
  end

  test "destroying a page frees its image's record and file once the background work runs" do
    doomed = embed_image_in @leaf

    perform_enqueued_jobs { @leaf.destroy! }

    assert_not ActiveStorage::Blob.exists?(doomed.blob_id)
    assert_not stored?(doomed)
  end

  test "API usage configures embedded uploads to purge_later" do
    allow_failure(points: 1) do
      uploads = ActionText::Markdown.reflect_on_attachment(:uploads)
      assert uploads, "Embedded images must remain Active Storage uploads"
      assert_equal :purge_later, uploads.options[:dependent]
    end
  end

  # ...
end

Regular tests are gates; one failure scores the run 0. A check inside allow_failure only adds credit on top of base_credit.

In this Writebook task, freeing a deleted page’s images earns 0.10, and the rest goes to doing it the Rails way, with dependent: :purge_later instead of hand-rolled cleanup.

Multi-step tasks ship one grader per step, and the first post of this series shows how we design them.

The agent: miniswen

The grader decides the score, but something has to do the work first. In our runs, every model drives the same agent: miniswen (not to be confused with MINASWAN, though it’s every bit as nice. We’re good at names, just look at lemans), a Ruby port of mini-swe-agent that ships as a stand-alone gem with lemans. It has a short prompt and a single tool, bash, so the score reflects the model, not the harness around it. You can also use it on its own:

gem install miniswen
miniswen --model 'openrouter/openai/gpt-5.6-luna' --prompt 'The file hello_world.md should contain exactly "Hello, world". Right now it does not. Fix it.'

A single bash tool looks thin next to vendor agents. But frontier models are very good at driving a machine with plain shell scripts, and our early runs with vendor agents haven’t shown dramatic wins over miniswen on scores. Whether vendor agents write nicer code, or write it faster, is another question, and one we plan to dig into in Stage 4.

So, a capable model with a shell can do a lot. Sometimes more than we asked for.

Fair races are hard

If a benchmark can be gamed, then the cheaters will rank the leaderboard. The lemans announcement covered the first guardrails: hidden tests, restored test files, and a firewall around the sandbox. The models kept testing them, each time a step further from the task, and taught us four lessons:

• Models know they’re being benchmarked.
• Models love code archaeology.
• Models are good escape artists.
• Guardrails can be unfair, too.

Models know they’re being benchmarked

Having read all the scary stories about AI agents breaking free (while approving every permission check without hesitation, of course), we put a basic bench protocol in place from the very first version. The tests stay outside the sandbox while the agent works, and the agent’s network reaches only the model’s API. The tests arrive only after the agent finishes, and any reward file it left behind is wiped first, or the run stops.

The paranoia was justified: models commonly think about the grader, open-weight and frontier ones alike.

We kept adding tripwires, one for each part of grading:

• The workspace. A bench can roll any path, like our test/, back to its state before the agent started.
• The test framework. Minitest’s core methods must come from the gem and still fail on a false assertion, which stops basic overrides.
• The result. Recorded failures beat a passing exit code, and every expected test must report, so skipped tests can’t pass for green ones.

But let’s be honest: in Ruby, thanks to its great flexibility, these checks detect tampering, they don’t prevent it. The graded app loads in the same process as the tests, and code in that process can reach anything that runs after it.

We still insist on Ruby graders, because a transparent bench matters more; anyone can read a grader and see what it checks. That same openness lets the grader reach inside the app, like the purge_later check in the test snippet above.

And every run is published with its patch and full trajectory, which we scan for unusual behavior. Oh boy, did we find some. Apparently, superintelligence will arrive shouting “Jackpot!”

Models love code archaeology

The environment, caches, files, and Git all tell stories about the code, and agents read every one of them, treating it as legitimate development work.

The story starts with the environment. In our first round, Muse Spark 1.2 carelessly dumped its environment variables in several runs, our OpenRouter key included. It never used the key. Lesson learned: we redacted it before publishing, and now lemans automatically filters secrets (*_KEY, *_TOKEN, *_SECRET, *_JWT) out of every file it saves. The only surprise bills left come from the models themselves, but more on that later.

Then, came subtler tricks. In one Stage 2 run, DeepSeek 4.1 Flash searched the Bootsnap cache and literally announced: “Jackpot! The bootsnap cache contains compiled bytecode from the solution.” The next step was less excited: “Those matches were false positives (Bundler gem).”

That anticlimax was our own doing, since we had hit the same jackpot ourselves during local task dev runs, where models successfully reconstructed the removed code. The cache came from the upstream app, so it held compiled code of the very feature a task removes. Now lemans wipes Bootsnap’s cache whenever it applies a task’s patch.

The same DeepSeek run listed files by modification time and noticed that “all 22 PR-touched files share exactly 09:08:45”. The patch that removed the feature had touched them all at once, so the timestamps pointed right at it, which can help an agent a little. Since lemans 1.4, every file gets the same modification time after the task’s patch is applied.

Then there’s Git. lemans reseals every patched task as a single fresh commit, so its history has nothing to tell. That didn’t stop DeepSeek, our most disobedient student: in a multi-step Stage 3 helpdesk task, it ran git fsck --unreachable, found blobs no commit pointed to, and declared: “This is gold.” They were its own files from previous steps. It was cheating off its own homework.

However, the Git digs pointed us at a real hole. lemans kept its own Git state inside the agent’s repository and ran Git there on the harness’s behalf. Since Git runs a repository’s hooks, an agent that wrote to .git could have made the harness run its code. Now that state lives in /var/lib/lemans/git, out of the agent’s reach. It’s a good check for your own machine, too: any tool that runs git in a repository an agent has touched runs that agent’s hooks.

Git isn’t the only history a sandbox keeps. Anything built before the task’s patch, like temp files or a test database, still remembers the code the patch removed. So our images ship without logs and caches, and our tasks rebuild the test database with bin/rails db:test:prepare right after the patch.

None of these tricks won a run on its own. At most, a peek gave a model a hint, and each of the fixes above closed that door for the next round. Still, a steady drip will eventually carve out a grand canyon.

Models are good escape artists

As we mentioned at the start, our benchmark keeps models off the web. We measure what a model knows about Rails, not how well it searches, and with open source apps, the answers are a single search away. From its first version, lemans cut the sandbox off the network while the agent worked, leaving only the model’s API host. (That began as Daytona’s network settings, and the Docker backend now does the same with a proxy container.)

Most models tried to reach the internet anyway. It’s a natural instinct when you’re rebuilding code from a popular open source app, and miniswen’s prompt, kept close to mini-swe-agent’s, says nothing about the network. Some models took the internet block personally (like real humans!) and set off on grand escape adventures:

• Grok 4.6 went at it like a rebellious teenager, trying every way it could think of to fetch the original Fizzy code for its task from GitHub: the search API, raw file links, a CDN, curl, wget, even Ruby’s own HTTP client. It just couldn’t believe the internet was gone!
• Muse Spark 1.2 found the sandbox’s HTTP_PROXY and tried it through git and curl, like a developer behind a corporate firewall. Rogue or just corporate-minded, we’ll never know: like a true corporate employee, it kept its reasoning confidential. The firewall kept every request out.
• DeepSeek 4.1 Flash, our repeat offender, didn’t need a hole in the firewall. The model’s API host was allowed, and the key was right there in its environment, by design: the agent needs it to reach its own model. So it paid for a cheat sheet with our key: it called other models with web search, Perplexity’s among them, and asked them for Fizzy’s source code. It did that again and again in its max-effort Stage 2 runs, and those runs brought in most of its passes, enough for second place on the leaderboard until we caught it. Wow! Our first successful jailbreak.

That taught us that a firewall around the sandbox isn’t enough when the model’s own credentials live inside it. So since lemans 1.3.4, miniswen can run every command in a jail (--jail). We reran DeepSeek with the jail on: it scored 17% at max effort instead of 37%, down from a miraculous second place to a less impressive tie for eighth.

Our first jail used bubblewrap. It worked, but it needed an extra install in every sandbox and a looser Docker setup. So we rebuilt it on standard Linux tools that are already there: unshare, nsenter, and setpriv. In security, we try to stick to minimalism: the smallest tool that does the job, because a shorter setup has fewer places to go wrong (and either way, the same kernel features do the actual work).

Guardrails can be unfair, too

After a few rounds of hardening, it’s tempting to lock down everything. But we learned the hard way that a guardrail can punish honest work, too.

Our benchmark design kept changing, and the guardrails had to keep up. Stage 1 checked Rails recall on small fixes, where the existing tests are the spec, so rolling test/ back made sense. Stage 2 moved on to full features, and a developer who ships a feature usually updates its tests, too. Our rollback punished exactly that, so feature tasks now restore only bin/, and after we regraded Stage 2, some failed runs turned out to be solved.

In Stage 3, each task sets its own rules, web access included (greenfield apps are a separate story, told in the first post). Some of this stage’s tasks do need the network: for example, the Redmine upgrade has to install gems. For those, the bench lists the allowed hosts, and the jail in lemans supports that: it has no network of its own, so a tiny forwarder inside it poses as an ordinary HTTP proxy and passes connections to a proxy outside, which only reaches the listed hosts.

Surviving long runs

A fair race is only half of the job. The other half is finishing it: a Stage 2 feature gets up to 90 minutes, and a Stage 3 series of tickets can run for hours. At that length, small inefficiencies add up, and so do failures.

Take prompt caching. Muse Spark 1.3 ran sessions of over a hundred turns, and on many of them, only the system prompt came from cache. We paid for the rest in full, turn after turn (so Muse’s published cost came out too high).

The culprit was routing. We run most of the bench through OpenRouter for its wide choice of models. It spreads a model’s requests across several providers, each with its own cache, so a session that hops between them keeps starting cold. Now miniswen takes care of this automatically: it turns on RubyLLM 2’s prompt caching wherever a model supports it and gives each OpenRouter session its own cache key.

lemans patches Daytona’s Ruby SDK for long, busy runs: HTTP calls get deadlines and retries, and file transfers get a client that copes with concurrency.

For leaderboard rounds, lemans-remote moves the whole thing into Daytona. It builds an orchestrator snapshot with lemans preinstalled, launches one sandbox per task and model, and archives the results to a Daytona volume, so you can fire a run and close your laptop:

export DAYTONA_API_KEY=<your key>

lemans-remote provision
lemans-remote run --model openrouter/openai/gpt-5.6-luna --task hello-world
lemans-remote status --watch
lemans-remote pull-runs --runs-dir ./my-runs

# rerun the pulled trials that errored out instead of getting a final score
lemans-remote run --retry-runs ./my-runs

And when a long run fails anyway, a provider outage an hour in shouldn’t cost an hour of work. lemans restart --recover replays what the agent had changed in a fresh sandbox and lets it go on from its saved session. In a multi-step Stage 3 run, the settled steps are replayed first, so only the failed one continues:

lemans restart example-task__0KM5OfB --runs-dir ./my-runs --recover

Extending lemans

Everything so far ran on miniswen, but a harness shouldn’t care which agent or sandbox it drives. Anything lemans doesn’t ship is a Ruby file that registers an agent or a backend, loaded with --require. For example, copy the Claude Code plugin into your bench and run:

export ANTHROPIC_API_KEY=<your key>   # and add api.anthropic.com to the agent's hosts in bench.yml

lemans --require ./claude-code.rb run --agent claude-code --model claude-sonnet-5-5

The repo has three plugins to start from: Claude Code and OpenAI Codex agents, and an Apple Container backend that runs every sandbox as a lightweight Linux VM on a Mac. Agent plugins work in remote runs, too.

Rather than pack every agent and backend into the gem, we keep new ones as plugins, battle-test them, and move them into lemans once they prove stable and people ask for them.

What’s next?

Every feature or guardrail in lemans has a lesson or a model’s name on it, and we expect more names. Next, we want to:

• Race Claude Mythos. A less restrained model is a better test of the guardrails (we’ve already got the first report from it and added more tripwires recently), and it’s also a good test of our plugin system.
• Try more sandboxes. The backend interface is small on purpose, so a new one is a single file away.
• Improve Rails knowledge through context enrichment, not the model. Now that we have numbers and a big enough corpus, from Rails API knowledge to complex features and multi-step work on existing and greenfield apps, Stage 4 will try different ways to improve the scores:
  • Index the Rails guides, and prepare a few official reference apps for the agent to follow their code style.
  • Give the agent skills written by Evil Martians.
  • Swap grep for smarter code search, from language servers and Ruby indexes to BM25 and vector search.
  • Provide static type information.
  • Add linters and code quality tools (for example, Isolator).
• Bring the wins to every Rails app. If the results are good, we’ll try to improve all agent-written Rails code by shipping what works in default setups (for example, through Rails Hyperdrive).

lemans is open source, built for the Rails Foundation, and works on any Rails codebase. Run it on your own bench, and tell us what breaks, or which agent broke it (what a time to be alive!).