The Agents on Rails Files: designing black-box evals for coding agents

The Agents on Rails benchmark is organized in stages: first, we evaluated models using atomic API-level tasks (Stage 1); then, we tested how well models can handle single feature tickets (Stage 2). The recently released Stage 3 goes further and asks whether a model can build on its own work: each task is a multi-step product or maintenance story, and each step is a new session working with whatever the agent left behind.

For Stage 1 and Stage 2, the instructions set code-level contracts, so we can fully rely on Rails unit and integration tests for verification. Most Stage 3 tasks, however, give agents much more freedom and don’t limit LLM creativity in code organization. Some of them are greenfield tasks where agents must implement full-featured applications from scratch by following product-level instructions. You can’t write Rails-level tests for a UI you’ve never seen plus the user actions associated with it. In this case, the only possible verification is one that treats the resulting piece of software as a black box and tests it from the outside, either using a web browser or HTTP calls (if there’s an API contract).

End-to-end testing (e2e), or black-box testing, has existed for ages. What changes when we apply it to LLM evals? The output is non-deterministic. So, when you ask an agent to build a feature, especially in a greenfield app, you rarely describe all the UI/UX quirks and label all the buttons, form inputs, and so on. Different models, even different sessions may produce distinct valid interfaces and user flows (within the user-defined requirements).

To reflect real-life prompting scenarios, evals must tolerate non-determinism and give agents enough freedom to demonstrate their capabilities.

In this post, I want to share the whys and hows behind our approach to designing black-box evals, based on two tasks from the Stage 3 batch: reimplementing chat bots for Campfire and building a conference live quiz application (Quizzy) from scratch.

Campfire chat bots, revisited

The eval in question, Campfire Bots, follows an already-classic pattern: picking an existing feature that has taken shape over years across multiple PRs, removing it from the codebase, and asking an agent to build it again. The evolution property of a feature (“multiple PRs”) was a must for us due to the multi-step task requirement. Chat bots fit that perfectly.

One benefit of this erase/replace design is that the upstream implementation can be used as a golden reference (and we always need one to prove our verification suite works). However, there is a caveat: Campfire has been public for long enough to be part of the training data. Indeed, almost all models recalled the codebase right away (“I’m recalling how Basecamp’s Campfire (ONCE) implemented bot users”).

That’s why our task spec deviates from the upstream. There are a few minor differences and one major change: every user is allowed to add bots, not only administrators. This significantly changes the security surface of the task. If you let anyone add programmable integrations to your product, expect misbehaving or even malicious actors to appear and be prepared (detect suspicious activity and ban bad actors). For this post, the bigger consequence is that the changes led to modifications in the UI/UX never present in the upstream.

Thus, we had to reconsider our verification suite for the task.

From white-box to black-box checks

In early versions of the task, we used Campfire’s own Rails test environment and hidden integration tests (mostly request tests) to grade the agent’s work. The only code-level contract in the instructions was a Rails console entry point for admins. That held up for the first trials, but it started to crack once we added steps that turned regular users into gods of bots.

Adding controller-level or route-level guidance to the ticket wasn’t an option. Stage 3 instructions aren’t senior-to-junior tickets with every nut and bolt specified. Rather, they only define product requirements.

If an instruction doesn’t tell you how to organize your models and controllers, the verification should not test them.

Our new security requirements also didn’t play well with the Rails test environment. Some mechanisms (such as log filtering) can be enabled only in production, and that’s legit. Thus, we needed to run a real server in a close-to-production mode and verify its behavior.

Finally, we saw an interesting opportunity because of codebase-agnostic verification. Campfire clones in other languages started to appear, e.g., Campfire in Rust. These clones are vibe-identical to the original Campfire: they produce the same UI and follow the same UX patterns. Thus, we could reuse our eval to see how the stack affects trial results. Same instruction, same verification suite—zero additional work to conduct stack battles!

So the checks had to move above the stack, into a web browser. However, that shift turned out to be challenging.

A poorly configured benchmark tests itself, not the code it’s supposed to test.

We picked a pretty standard Ruby stack: Minitest, Capybara, Cuprite, and headless Chromium. The initial version of the e2e suite was a direct port of the existing Rails test suite and looked more or less like typical Rails system tests, except for custom web server management. We needed to run the app in a standalone process due to production-like environment requirements.

The original port was barely usable: flaky, tightly coupled to the solution’s markup, and slow: minutes instead of seconds. Designing an eval takes many iterations, so these extra minutes can turn into hours lost to a broken harness that reports invalid results.

Thus, we took a half-step back and created a third, hybrid version of the task:

• The same instructions as the fully end-to-end version plus a couple of code-level hints (Rails console commands).
• The same oracle (golden solution).
• Verification mostly inside the app, so the checks can call real code, plus a few integration-level checks.

This hybrid version allowed us to shorten the feedback loop while building the black-box verification suite. We ran trials multiple times (a golden solution plus a couple of real models) and collected per-step patches and grading results. Then, we replayed the same scenarios with browser-based tests and evaluated our e2e harness using the following rule: a trial that passes the hybrid task must pass the e2e version, unless the e2e version found a real bug. Otherwise, we (ask Claude to) update the failing check or test helper to recognize a newly discovered pattern.

In the end, we built an e2e suite that agreed with the inside-app verification suite on every initial trial. With that, we were ready to welcome more models to the party!

On calibration and mutation

After a couple of trials with new models, we quickly found that our browser-based verification wasn’t robust enough: agents can be very creative in implementing UI. After a handful of trials failed due to our overly strict harness, we stopped brute-forcing and decided to try a different strategy for identifying the gaps in our e2e tests: mutation testing.

Instead of collecting new patches through long-running trials with different agents, we synthesized new versions of the app using mutants of two kinds:

• Equivalent mutants change the UI but not the behavior. Every test must still pass.
• Faulty mutants break the behavior behind a UI that looks right. At least one test must fail.

For the Campfire Bots task, we’ve ended up with 13 mutants (8 + 5). Here are some examples:

Equivalent mutants mostly focus on UI tweaks and ensure our harness is robust enough to fairly grade different HTML styles within the task requirements. Faulty mutants, on the other hand, check that the grader actually catches a broken requirement.

Mutation testing helped us build the e2e verification suite that successfully and fairly evaluated dozens of trials on the first try. Only 2 of the 33 runs caused the grader recalibration and the subsequent re-verification.

Black-box evals for apps you’ve never seen

Campfire Bots paved the way for more end-to-end evals. We had designed the method; we had identified common testing patterns and tricks. We were ready to apply this knowledge to the next challenge, which is a task that involves building a full-featured application from scratch.

For the Campfire task, we could rely on the baseline app features and UIs in our tests: authentication flow, existing screens and so forth. Yes, agents still produced random UI bits for the Bots feature, but the rest of the app stayed the same and known to us. When you ask an agent to build something from scratch, how do you pick the stability points for your verification suite?

1. Option A is to provide a detailed spec as part of the instruction with many testable anchors: HTTP paths, labels, form field names, and so on. Does it sound like a product ticket? Not really. We can expect some anchors, e.g., accessibility, to be defined in the instruction but not everything.

2. Option B is to provide ready-to-use mockups and ask an agent to follow them closely. Most design tools support exporting mockups in HTML/CSS format, so that could become a legend for the task in question.

3. Option C is to use AI-driven browser automation instead of deterministic tests (for example, Browser Use). Going from code to text for describing scenarios means that now we have to tune prompts and somehow verify their correctness and fairness first. Both trials and replays now cost money. It’s probably worth it as a last resort but not as a go-to solution.

For the Quizzy task—building a live quiz app from scratch, we ended up with a combination of A and B:

1. Added QA requirements with accessibility hints to the instructions. They normalize the UI just enough for tests to find things, for example: “Each item of a list sits in its own list item or table row”, “Switches are checkboxes”, “Confirmations use the browser’s native confirm() dialog”.
2. Provided example mockups in HTML/CSS format. Just a couple of screens, not everything to be implemented.

That gave us a good balance as the instructions aren’t over-specified, and verification stays deterministic.

After repeating the same mutation-calibration phase, we ended up with a verification harness that could reliably grade diverse implementations. Look at these versions of the same Quizzy screen—all from passing trials!

And let’s look at the code of the e2e tests to see how we managed to use deterministic checks for non-deterministic outputs.

E2E test hacks and tips

As we mentioned, the e2e tests were written by AI under human supervision. In a sea of barely-readable XPath selectors and pessimistic finders that should only be read by AI, we can find some techniques worth our human attention. Let me share some tricks from our black-box test suites.

The #submit_and_wait helper

def submit_and_wait(timeout: 5)
  network = page.driver.browser.network
  network.clear(:traffic)
  yield
  eventually(timeout:, interval: 0.1) { network.traffic.any? { |e| e.request&.method == "POST" && !e.pending? } }
end

# Example usage
submit_and_wait { click_on "Save changes", match: :first }
assert_text "New name"
visit "/bots"
# ...

This helper wraps a form-submitting action and ensures that we proceed only after a request has completed. That guarantees that the request hasn’t been canceled by the subsequent navigation. Having a content assertion after the submission (assert_text "...") may be not enough in case of optimistic UI updates.

The #eventually helper

def eventually(timeout: 15, interval: 0.25)
  deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
  loop do
    result = yield
    return result if result
    return result if Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline
    sleep interval
  end
end

This is one of the most popular helpers in the test suites (and I bet your browser-based tests have something similar). Yes, wait-try-repeat is a common pattern when you have no other deterministic wait anchors to rely on.

Testing authentication with traffic replay

Generated greenfield apps are free to choose any authentication strategy and method. How do you verify that authentication is enforced? We can do that by recording traffic first for the authenticated session and then replaying it within an anonymous session and checking that requests fail:

test "moderator actions need the moderator's credentials" do
  url = as_moderator { start_game(title) }
  as_player("Mallory") { visit url }

  recorded = capture_writes { moderator_click "Start" }
  refute_empty recorded

  replays = as_player("Alice") { recorded.map { replay(it, stranger: true) } }

  accepted = replays.reject { (400..499).cover?(it["status"]) && it["status"] != 422 }
  assert_empty accepted

  # ...
end

def capture_writes
  page.driver.browser.network.clear(:traffic)
  yield
  finish_pending_posts
  page.driver.browser.network.traffic.filter_map do |exchange|
    request = exchange.request
    next unless request && %w[Fetch XHR Document].include?(request.type)
    raw = request.to_h["request"]
    next if raw["method"] == "GET"
    { method: raw["method"], url: raw["url"], headers: raw["headers"] || {}, body: raw["postData"] }
  end
end

def replay(record, **)
  # Sends a recorded write as the current session (cookies, CSRF, and such)
end

Emulating offline conditions with support for WebSockets

Traffic control is possible over CDP. You can emulate various network conditions, including no network at all (“offline”). However, just like in Chrome DevTools, turning the “offline” mode on with CDP doesn’t close active persistent connections, such as WebSockets. The only workaround is to track WebSocket connections yourself and close them manually:

# Patching new WebSocket(...)
page.driver.browser.page.command("Page.addScriptToEvaluateOnNewDocument", source: <<~JS)
  const NativeWebSocket = window.WebSocket
  window.__sockets = []
  window.WebSocket = class extends NativeWebSocket {
    constructor(...args) { super(...args); window.__sockets.push(this) }
  }
JS

Now, we can test the client recovery under unreliable network conditions:

test "participant catches up after losing connection" do
  run = @quiz.runs.create!
  join(:alice, run, "Alice")

  using_session(:alice) do
    # set up the patch
    page.driver.browser.page.command("...")

    visit play_path(run)
    assert_selector "turbo-cable-stream-source[connected]"

    page.driver.browser.network.emulate_network_conditions(offline: true)
    page.execute_script("window.__sockets.forEach((ws) => ws.close())")
  end

  # The moderator starts the quiz while the phone is offline
  run.start!

  using_session(:alice) do
    assert_no_selector ".play-question__body"
    page.driver.browser.network.emulate_network_conditions(offline: false)
    # No manual reload: the client reconnects, restores the session and receives the missed refresh
    assert_selector ".play-question__body", text: "Question 1?", wait: 15
  end
end

The --disable-back-forward-cache switch

This workaround is also WebSockets-related: a navigation can keep the old page frozen in the cache with its WebSocket open, so the server never sees the socket disconnected (and keeps the participant “online” in the Quizzy app).

Watch for timeouts, human

The last trick hints that reading the generated code is still worth it. Here’s why:

A typical workaround to make browser-based tests more reliable is to increase timeouts or sprinkle sleep-s here and there. We humans used to accept this lesser evil (aka technical debt) for the sake of green CI builds. Now, LLMs trained on our legacy tend to generate very defensive tests with ridiculously high timeouts and a lot of sleeping, thus, making the verification phase needlessly slower. For example, the original #submit_and_wait helper generated by AI had default timeout of 30s; meaning half a minute wasted every time the check fails! A web server running locally can’t be that slow to wait for more than 5s.

Give yourself a minute and skim through the test configuration and helpers and tune timeouts to reflect the software under test, not just overestimated assumptions made by a “smart” slot machine.

Bonus: rewrite it in Rust, or look at all the things agents aren’t doing with Rails

Having black-box evals ready and battle-tested for Campfire Bots and Quizzy, we couldn’t resist the temptation to reuse them for stack-to-stack comparisons. Which stack is better for greenfield applications? Which version of Campfire, the original Rails or the Rust vibe-port, is easier to extend with new features?

Let’s start with the Campfire Bots task.

We picked the following models: Claude Opus 5.5, GPT-6 Astra, Muse Spark 1.3 and Gemini Flash 3.8. That’s two top-tier and two mid-tier models by the Agents on Rails leaderboard at the time. Opus and Astra solved both stacks. Muse and Gemini failed on both stacks. The resulting score was same-ish for all passed runs: the stack didn’t define it. The metrics, however, have something to say:

Rust cost more for the same models in both dollars and minutes. Most of the extra cost was reading: input tokens grew 3–7x; output tokens grew only 1.5–2.1x. Why? The port itself is the main cause. The Rust app is an AI-generated, Rails-shaped port: ~13x more code than the Rails app, a rebuilt Rails framework, and Rails idioms uncommon in Rust-native applications. Agents had to learn not only the application but the whole framework from scratch!

Building an app from scratch is a different story. Look at the Quizzy trial results:

Frontier models are stack-agnostic. Opus’s best app is in Rails (1.00); Astra’s best is in Rust (1.00). Memory guarantees and types don’t seem to matter much here.

Cheaper models work well with a strong framework. GPT-6 Luna and DeepSeek V4.1 Flash solved the Rails version but not the Rust one. In Rust, the agent must invent and remember every technical concern itself: HTTP headers, real-time updates, and so on. In Rails, you get all of this by default—batteries are included! All the things agents aren’t doing when with Rails cost them nothing and result in production-ready software.

One important signal is missing, though: the UI/UX perceived quality. Machines can’t really grade it, but you can. Look at the video below and guess which application is a Rails trial and which one is a Rust trial.

Recap: a checklist for black-box evals

Today, whenever we work on black-box evals like the ones in this post, we follow the same guidelines:

• Specify the product, not the code. Keep your instruction as close to a product-level ticket written by a tech lead or a senior/QA engineer as possible.
• Start with a faster white-box twin if possible. Faster feedback loop is still one of the most important metrics.
• Use mutation testing to find verification gaps. Start with two types of mutants: equivalent mutants must pass; faulty mutants must fail.
• Calibrate on models that fail differently. Pick diverse models for the initial trials, learn from the failures, improve the harness, repeat.
• Grade in a production-like environment. The test environment is too relaxed for a browser-driven run.

Now that code is cheap, the processes and methodologies become more important.

Just as we moved from white-box to black-box testing in our evals, the role of a software engineer shifts in the same upward direction, from engineering code to engineering platforms, harnesses, eval designs, and all the things we didn’t even think about a mere couple of years ago.

That means there’ll be more ideas and tricks to share in our blog and more work to accomplish.

Follow us for more behind-the-scenes insights on the Agents on Rails benchmark project and share your AI-inspired challenges!