A fixture-based approach to interface testing in Rails

May 27, 2019

Topics

Andrew Kozin
Backend Engineer

The problem

In the past few months, my team has been working on a big project written with Ruby on Rails. The key component of our application is integration to many external APIs like different eBay services, postal services, logistics operators, translation services, etc. All the interfaces we use are complex and rich. Most of them are refined continuously: new versions are released from time to time, while the legacy ones stop being supported. Generally, for projects this complex, you should expect neither full nor 100% consistent documentation coverage.

The whole picture became even more complex when we turned from external services to “internal” ones that lay between frontend and backend parts of the application. Some time ago, we decided to switch from RESTful API to GraphQL here, and our life as developers became happier. But there’s no honey without bees. The bee-ness of GraphQL interface is the backside of its flexibility. And the more flexible the API became, the more edge cases we had to take into account and cover by tests.

That’s why on the backend we rely on our own tests, trying to cover as many corner cases as we can. Sometimes the specs are the only evidence of how the program was intended to work. Even the author of a feature forgets details over time. But it’s a bad sign when a spec has to be read by colleagues, whether they are code reviewers or new members of the team. That’s why the readability of tests matters and is worth the effort.

But what does it mean for a test to be readable? I believe that it should not only cover edge cases properly but tell a story about our expectations. It must do this step-by-step, without overloading a reader with details. Ideally, it should show the big picture of what we expect, and this declaration must be separate from how we model the expectation.

Skipping ahead, I would like my test to look something like this:

RSpec.describe ProductTranslator, ".call" do
  subject { described_class.call(listing) }

  before do
    stub_fixture "#{__dir__}/#{edge_case}/stubs.yml"
    seed_fixture "#{__dir__}/#{edge_case}/database.yml", listing_id: 42
  end

  let(:listing) { Listing.find(42) }
  let(:target)  { load_fixture "#{__dir__}/output.yml" }

  it "updates the listing with translated specifics" do
    expect { subject }.to change { listing.reload.specifics }.to target
  end
end

In the rest of the post, I’ll try to show our fixture-based approach to making tests understandable, and small tools we use for this goal. Because we use the Rails-based application, I refer to the specific tools from its ecosystem (RSpec, FactoryBot, and the Database Cleaner). But the whole idea of this post is framework-agnostic: use fixtures to prepare as much context for the specification as you can.

Web API as a test subject

Before going further, let’s look at the APIs as a specific subject of the test.

From the consumer’s point of view, all public web interfaces are alike. Following its business logic, the application prepares a request and sends it to the server. The server responds with data that must be processed it one way or another, depending on the content of the response. Most interfaces use text-based formats (JSON, XML, GraphQL) for data serialization.

Here we should specify two parts of the consumer’s logic:

the correctness of request preparation depending on the internal state of the app,
the correctness of response processing, which depends on both the response and the internal state of the application.

To be more specific, we use three different specs:

unit tests for request builders,
unit tests for response handlers,
point-to-point test for the whole service doing the request.

Sometimes it makes sense to specify under-documented APIs as well. The goal is not to test a remote server, but ensure the correctness of our own expectations about its behavior. For simplicity, I keep this task aside. Those who interested in the corresponding techniques can check Blood Contracts by my colleague Sara Dolgan.

When testing our own application, we must stub the remote interfaces because they are out of our control. This is an important lesson I’ve learned over the years: use real responses obtained “in vivo” for stubbing! Learning this lesson was not easy for me. Many times when I used examples “in vitro” from docs, it led to hours of debugging. The rule of thumb is “stub APIs based on real data.”

To summarize, the context of our tests include:

DB seeds and stubs to provide the internal state of the application,
stubs of our client to the external API,
real examples of requests/responses (possibly mutated for various edge cases).

Fixtures take the stage

First, let’s look at an example. This is a specification of a class that translates nested data like { name: "Color", value: "Red" } using the GoogleTranslateDiff client to Google Translate API. While we hide HTTPS requests and responses behind the client interface, this wrapper is thin enough to accept plain text along with locales, and return a plain text.

# spec/services/product_translator/_spec.rb

RSpec.describe ProductTranslator, ".call" do
  subject { described_class.call(listing) }

  before do
    # Stub the client to remote API
    allow(GoogleTranslateDiff)
      .to receive_message_chain(:translate)
      .with do |text, from:, to:|
        case [text, from, to]
        when %w[Color en ru] then "Цвет"
        when %w[Black en ru] then "Чёрный"
        when %w[Brand en ru] then "Марка"
        when ["<span class='notranslate'>Apple</span>", "en", "ru"]
          "<span class='notranslate'>Apple</span>"
        else raise
        end
      end
  end

  # Seed the necessary objects in the database
  let(:listing) { FactoryBot.create :listing, source: product, locale: target }

  let(:product) do
    FactoryBot.create :product, :ready, locale: :en, specifics: input
  end

  # Prepare data structures for input/output
  let(:input) do
    [{ name: "Color", value: "Black" }, { name: "Brand", value: "##Apple##" }]
  end

  let(:output) do
    [{ name: "Цвет", value: "Чёрный" }, { name: "Марка", value: "Apple" }]
  end

  context "when target locale differs from the product's one" do
    let(:target) { "ru" }

    it "updates the listing with translated specifics" do
      expect { subject }.to change { listing.reload.specifics }.to output
    end
  end

  context "when target locale is the same as the product's one" do
    let(:target) { "en" }

    it "updates the listing with the original specifics from the product" do
      expect(GoogleTranslateDiff).not_to receive(:new)
      expect { subject }.not_to change { listng.reload.attributes }
    end
  end
end

As you can see, even a simple test can become difficult to read when there are too many details.

In complex cases (when you should vary both the internal state and responses to cover all corner cases) a whole specification turns into spaghetti. But there’s good news. As I mentioned, both the request and response are just text-formatted in some way (JSON, XML, GraphQL, etc.). As a rule, our clients deserialize those data into JSON-compatible basic Ruby objects, not application-specific classes.

That’s why we can easily extract them into fixtures like this:

# spec/services/product_translator/output.yml
---
- name:  Цвет
  value: Чёрный
- name:  Марка
  value: Apple

Now we can simplify our specification a bit:

let(:output) { YAML.load_file "#{__dir__}/output.yml" }

The specification became better, but not much better. We still have stubs and seeds in it. What’s more important is that after hiding some structures in fixtures, the model expectations became distributed between the fixture and the test, and the observability was lost. For example, when reading the specification, it’s unclear why we stub the client with ‘Color’, ‘Black’, ‘Brand’, etc. What role do they play?

Wouldn’t be better if we could move all the specific parts, including stubbing and seeds, into fixture layer, and make the specification just a clue? If we could move all the details of method stubbing, DB seed, and data serialization into fixtures?

But we can! Meet the Fixturama, a kind of “fixture on steroids.” The gem was extracted from our project and has been heavily used in production for several months. You can look at the details of usage in a README. Here I’ll show how its usage can improve our specification.

In the next section, I’ll show you several simple techniques supported by the gem.

Fixtures on steroids

To add helpers to your suite, just add the dependency:

require "fixturama/rspec"

Stubbing

Now we can start with stubbing. In a YAML file you can define a set of opinionated keys:

# spec/services/product_translator/stubs.yml
---
- class: GoogleTranslateDiff
  chain:
    - new
    - translate
  arguments:
    - Color
    - :from: en
      :to: ru
  actions:
    - return: Цвет

- class: GoogleTranslateDiff
  chain:
    - new
    - translate
  arguments:
    - Brand
    - :from: en
      :to: ru
  actions:
    - return: Марка

- class: GoogleTranslateDiff
  chain:
    - new
    - translate
  arguments:
    - Black
    - :from: en
      :to: ru
  actions:
    - return: Чёрный

- class: GoogleTranslateDiff
  chain:
    - new
    - translate
  arguments:
    - "<span class='notranslate'>Apple</span>"
    - :from: en
      :to: ru
  actions:
    - return: "<span class='notranslate'>Apple</span>"

- class: GoogleTranslateDiff
  chain:
    - new
    - translate
  actions:
    - raise: StandardError

While the fixture got quite verbose (because we define separate action for every variance of arguments), the specification itself is now simple:

before { stub_fixture "#{__dir__}/stubs.yml" }

In some cases, you will collect all the stubs in one fixture, or you can split them between different specs (for various edge cases).

You can customize more options here to set different returns for consecutive calls of the method, etc. See more examples on GitHub.

Seeding

After moving stubs away, we can do the same for database objects. Here the fixturama provides a thin wrapper around the FactoryBot. Another opinionated syntax:

# ./database.yml
---
- type: product
  traits:
    - ready
  params:
    id: 1
    locale: en
    specifics:
      - name:  Color
        value: Black
      - name:  Brand
        value: '##Apple##'

- type: listing
  params:
    id: <%= listing_id %>
    source_id: 1 # bound to the product above
    locale: <%= target %>

In a specification you run the seed as:

before { seed_fixture "database.yml", listing_id: 42, target: "ru" }

let(:listing) { Listing.find(42) }

Now we’ve removed all the details from the specification. The only thing we need to tell a reader is: “there’s a listing with the same id as we used in a fixture.”

Without fixtures the database preparation would look like this:

let(:listing) { create :listing, source: product, locale: "ru" }
let(:product) do
  create :product, :ready, locale: "en", specifics: [
    { "name" => "Color", "value" => "Black" },
    { "name" => "Brand", "value" => "##Apple##" }
  ]
end

It may seem shorter than a fixture (unless you need huge jsonb structures inside). But remember the goal: I wanted to move all the details out of specs. If I moved stubs and results only and left database objects in the specification, I would end up with settings split between the test and its fixtures. This approach would make understanding what’s going on much harder for a reader.

With the “fixture-based” approach, we don’t even need to look at the specification. We can just compare the database.yml and seeds.yml with the output.yml they provide. The specification by itself just serves a clue between those fixtures.

Notice that under the hood, we use the ERB binding supported by all the fixturama helpers. You can use it in seeds, stubs, and loads in exactly the same way.

Traps and workarounds

Now I must stop and tell you about a trap. To refer different objects to each other inside a seed fixture, we use concrete ids. Let’s look at a more verbose example from another specification:

---
- type: product
  traits:
    - ready
  params:
    id: 1
    locale: en

- type: listing
  traits:
    - published
  params:
    id: <%= listing_id %>
    source_id: 1
    locale: <%= target %>

Here I just need a published listing of the ready product described in English. To bind them together, I hardcoded the id of the product, but this could create a problem with flaky, non-deterministic tests.

This problem occurs when you combine explicit ids with those generated by default by the database. In isolation, every spec can work fine, but running them in random order can lead to conflict. Of course, you could use the truncation strategy of the database cleaner, but this leads to a huge loss of performance.

Now we have a much better solution, both clean and elegant. The trick is to add the offset o DB sequences of ids so that they start iteration from a big number (say, 10'000'000). Now you can safely use smaller ids in fixtures; the database would never generate the same record identifiers by default.

In the fixturama you can use a special setting:

# ./rails_helper.rb
RSpec.configure do |config|
  config.before(:suite) { Fixturama.start_ids_from(10_000_000) }
end

Serialization

All we have left is slightly improve the serialization. Remember the original way of extracting fixtures:

let(:output) { YAML.load_file "#{__dir__}/output.yml" }

With the fixturama you could do the same using a loader:

let(:output) { load_fixture "#{__dir__}/output.yml" }

This is not a big improvement, but you don’t need to specify a format explicitly. Both the YAML and JSON-formatted files are serialized automagically; otherwise, the content is loaded as a plain text.

You can also bind values via ERB now; there are many cases when this option is useful.

Test organization

As you might have noticed, in all the examples above I used the __dir__ world to refer to the current folder, instead of placing fixtures into the special folder like spec/fixtures. Keeping a fixture close to the specification makes it much easier to understand and modify.

Typically, I create a folder for every specification and organize files like this:

specs
  services
    translation
      _spec.rb
      database.yml
      stubs.yml
      output.yml

For many edge cases, you can go deeper:

specs
  services
    translation
      _spec.rb
      valid_input
        database.yml
        stubs.yml
        output.yml
      invalid_input
        database.yml
        stubs.yml
        output.yml

Now all you need to cover a new edge case is just copy-paste a subfolder and modify its content.

The final example

Let’s look at the final example of how a specification looks after all these modifications:

require "fixturama/rspec"

RSpec.describe ProductTranslator, ".call" do
  subject { described_class.call(listing) }

  before do
    stub_fixture "#{__dir__}/#{edge_case}/stubs.yml"
    seed_fixture "#{__dir__}/#{edge_case}/database.yml", listing_id: 42
  end

  let(:listing) { Listing.find(42) }
  let(:output)  { load_fixture "#{__dir__}/#{edge_case}/output.yml" }

  context "when target locale differs from the product's one" do
    let(:edge_case) { "different_locales" }

    it "updates the listing with translated specifics" do
      expect { subject }.to change { listing.reload.specifics }.to output
    end
  end

  context "when target locale is the same as the product's one" do
    let(:edge_case) { "same_locales" }

    it "updates the listing with the original specifics from the product" do
      expect(GoogleTranslateDiff).not_to receive(:new)
      expect { subject }.not_to change { listng.reload.attributes }
    end
  end
end

The last change I would suggest here is to extract expectations into shared examples:

require "fixturama/rspec"

RSpec.describe ProductTranslator, ".call" do
  subject { described_class.call(listing) }

  # DEFINITIONS

  before do
    stub_fixture "#{__dir__}/#{edge_case}/stubs.yml"
    seed_fixture "#{__dir__}/#{edge_case}/database.yml", listing_id: 42
  end

  let(:listing) { Listing.find(42) }
  let(:output)  { load_fixture "#{__dir__}/#{edge_case}/output.yml" }

  # EXAMPLES

  shared_examples "translating specifics" do
    it "updates the listing with translated specifics" do
      expect { subject }.to change { listing.reload.specifics }.to output
    end
  end

  shared_examples "skipping translation" do
    it "doesn't call the remote API" do
      expect(GoogleTranslateDiff).not_to receive(:new)
      subject
    end

    it "updates the listing with the original specifics from the product" do
      expect { subject }.not_to change { listng.reload.attributes }
    end
  end

  # CONTEXTS

  context "when target locale differs from the product's one" do
    let(:edge_case) { "different_locales" }
    include_examples "translating specifics"
  end

  context "when target locale is the same as the product's one" do
    let(:edge_case) { "same_locales" }
    include_examples "skipping translation"
  end
end

All the details of how specs are prepared, and how examples are modeled, are isolated in fixtures and shared examples accordingly.

With this change, we can easily cover new edge cases with the following small steps:

add the new folder with corresponding fixtures,
add new context to the spec, making sure to select a folder and expected behavior.

Now the whole test will remain understandable and traceable even with dozens of corner cases covered.

Further improvements

There are more improvements and optimizations available. I’ll stop here and just refer to the gem test-prof by Vladimir Dementyev (aka @palkan) where you can find many ideas.

For example, if you need the same database objects for all edge cases, you can make the preparation a bit faster by changing before to before_all:

before_all { stub_fixture "#{__dir__}/database.yml" }

This time you will have the time-consuming task of creating the database objects only once. All in all, I highly recommend looking at the gem and using it in your specs, either fixture-based or not.

Some restrictions

I started this post with a discussion of API-specific problems but later explored how the fixturama could be a good solution for many specifications, not related to public interfaces. Sometimes this will be true; we use the gem for testing data mappers, some service objects, and policy objects not related to any external API. Unfortunately, YAML or JSON-serialized structures cannot cover all the necessary cases. Its suitability backs on the typical formats of the API requests and responses (text, JSON, YAML, GraphQL).

Another limitation arises from a possible irregularity of tests. When you have to cover many corner cases with the same structure (seed, stub, prepare the input and expected output), this approach works well. When you want to test different methods with heterogeneous preparation steps, it becomes less useful.

The third restriction has to do with the verbosity of the fixtures. It works well when you extract big structures from the test and is suitable when you can “program behavior” by comparing complex input.yml to output.yml, without having to care about the nuts and bolts of the spec. But if your structures are simple (like a short string on input compared to boolean on output), it converts to an overkill. That’s why in our real test suite we combine fixture-based specs with more traditional ones.

All that said, I still recommend you try to simplify specifications with the Fixturama and make your interaction with API more safe, predictable, and understandable for you and your team.

I’d love for you to give it a try, and leave your feedback on what can be done better, and how your test cases could be simplified further with additional settings.