# WebSocket Director: scenario-based integration tests for realtime apps

> This post introduces WebSocket Director (wsdirector), a testing automation tool for WebSocket servers which uses human-readable scenarios stored in YAML or JSON files.


- Date: 2022-09-20T00:00:00.000Z
- Authors: Vladimir Dementyev, Travis Turner
- Categories: Infrastructure, Real-time, DX
- URL: https://evilmartians.com/chronicles/websocket-director-scenario-based-integration-tests-for-real-time-apps

---

Black-box testing, end-to-end testing, or _system testing_, according to Ruby on Rails, is just the tip of the testing iceberg (or diamond, as you wish). Still, it plays a significant role achieving software stability. However, writing and maintaining high-level integration tests is not a piece of cake, especially when you go beyond the _classic_, synchronous HTTP request-response model. In my case, all the flavors of WebSockets. To help myself with everyday testing tasks, I've built a tool, [WebSocket Director][wsdirector], which I'm happy to introduce in this post.

Modern software development, especially the web part, places much more attention on engineer productivity than the compiler speed, language runtimes, or whatever. Today, **humans are the bottleneck**. And to eliminate this bottleneck, we need **developer tools**.

Have you ever tried counting the number of dev tools you use regularly? It doesn't matter; it's never enough. There's always a routine task which you'd prefer some _robot_ to do for you.

I'd like to share the story of one such dev tool, [WebSocket Director][wsdirector], and demonstrate how you can use it to increase your productivity.

## From fruitless search to Shakespeare

Another dev tool that came out of AnyCable development is [ACLI][acli]—Action Cable command-line interface. Unlike WebSocketDirector, ACLI aims to provide an interactive REPL-like experience for communicating over WebSockets via Action Cable protocol.

When I started working on [AnyCable][anycable] back in 2016, I struggled to find a black-box testing tool for WebSockets. I wanted something human-readable and human-editable, which I could repeatedly run in my terminal to simulate typical real-time activities. Either I wasn't good at searching, or my taste was too picky (yeah, productivity is subjective), but I couldn't find anything to satisfy my needs.

*Read also Evil Martians' article*: https://evilmartians.com/chronicles/anycable-actioncable-on-steroids

So, what exactly was I looking for? Here are some scenarios I wanted to automate:

- Connection handshake (create an HTTP connection, wait for handshake confirmation from a server).
- Subscribing to channels (or the `connect - subscribe - confirm` loop).
- Performing Action Cable actions.

To my Rubyish mind, the best way to formalize such scenarios seemed to use YAML. For example, the Action Cable scenario I initially sketched looked like this:

```yml
- subscribe:
    channel: "EchoChannel"
- perform:
    channel: "EchoChannel"
    data:
      text: "Hey!"
- receive:
    channel: "EchoChannel"
    data:
      response: "Hey!"
```

Readable? Yes. Functional? Sure. That's good enough for a basic smoke test when you work on a custom server _speaking_ [Action Cable protocol][action-cable-protocol].

I continued iterating on possible use cases and the corresponding scenarios without writing any code. That was intentional: I didn't want code decisions to affect the script's format; I intended it should stay _tool-agnostic_.

For real-time applications, it's crucial to perform multi-user tests. I borrowed the _scale factor_ concept from [pgbench][] and extended the original file structure to support multiple client groups. The resulting multi-user scenario looked like this:

```yml
- client:
    multiplier: ":scale"
    name: publishers
    actions:
      - subscribe:
          channel: ChatChannel
          params:
            room_id: "42"
      - wait_all # a special synchronization directive
      - perform:
          channel: ChatChannel
          params:
            room_id: "42"
          action: "speak"
          data:
            message: "test"
- client:
    multiplier: ":scale * 2"
    name: listeners
    actions:
      - subscribe:
          channel: ChatChannel
          params:
            room_id: "42"
      - wait_all
      - receive:
          multiplier: ":scale"
          channel: ChatChannel
          params:
            room_id: "42"
          data:
            message: "test"
```

Before jumping into implementation, the final touch was to come up with a good name for the tool to execute these scenarios. The phrase "executing scenarios" naturally led to "directing scripts" and... bingo! The name _WebSocket Director_ was born. And then, while using it, a developer becomes a _scriptwriter_. Writing automation scenarios is a creative process, so why not call a tool appropriately?

> All the world's a server, and all the men and women merely clients.

For WebSocket load testing, I prefer to use dedicated tools: [websocket-bench][] for AnyCable performance tests and [k6][] with [xk6-cable][] for application stress tests.

And the rest is history (you can see [on GitHub](https://github.com/palkan/wsdirector/commits/master) 😉). Today, I only need to run `gem install wsdirector-cli` to perform WebSockets tests (and even _mini-benchmarks_). After installation, I can _direct_ the above script like this:

```sh
$ wsdirector chat.yml localhost:8080/cable -s 10

Group publishers: 10 clients, 0 failures
Group listeners: 20 clients, 0 failures
```

But, despite what you might think, developing AnyCable is not the only use case for WebSocket Director. Let me share a couple of stories where WebSocket director really clicked into place.

## WebSocketDirector vs. Twilio Voice

*Read also Evil Martians' article*: https://evilmartians.com/chronicles/anycable-goes-pro-fast-websockets-for-ruby-at-scale

[Twilio][] is a powerful platform for building communication tools of any kind—from SMS, to voice messages. Their [Programmable Voice][twilio-voice] service provides a unique functionality, which I call _WebSocket hooks_ (referring to HTTP webhooks). You can configure your application to consume audio packets through a WebSocket connection in real-time (see, for example, [this tutorial](https://www.twilio.com/docs/voice/tutorials/consume-real-time-media-stream-using-websockets-python-and-flask)).

That could be useful, for instance, to perform some data analysis or to build AI auto-responders. But anyhow, the possibilities that Twilio offers are not the subject of this article. Today, we're talking about developer happiness (or the lack of it).

Imagine a development flow when working on a feature that involves processing voice streams:

- You start a web server (say, `rails s`).
- You start a tunneling service to expose your local server to the Internet (for example, Ngrok).
- You open a Twilio console and make a phone call using a web phone simulator.
- And repeat 'till the end of the work day.

Unlike HTTP webhooks, which you could just simulate with `curl`, WebSockets communication is not that easy to reproduce locally. It takes quite a bit of time to perform all the necessary steps; this kind of feedback loop contains more chores than actual work.

While working on such a project, I quickly found this development process anti-productive and started looking for optimizations. Then, I recalled WS Director: "What if we could create a fixture script for a Twilio call?"

All I needed was to capture the incoming WebSocket frames and transform them into WS Director _steps_: each message could be represented as a `send` step, and the intervals between messages could be emulated via the `sleep` steps. I'm not going to share the original implementation of the WebSockets VCR here; instead, let me demonstrate how to capture the incoming WebSocket stream with the latest version of WS Director and a slightly patched Action Cable server:

```ruby
module ApplicationCable
  class Connection < ActionCable::Connection::Base
    # We patch connection callbacks to simply consume messages
    # without entering the channels layer at all
    def handle_open
      # Snapshot is a container for frames,
      # which takes care of inserting `sleep`-s between messages
      @snapshot = WSDirector::Snapshot.new

      # This line is required to start receiving messages
      message_buffer.process!
    end

    def dispatch_websocket_message(data)
      @snapshot << decode(data)
    end

    def handle_close
      File.write("tmp/call.yml", @snapshot.to_yml)
    end
  end
end
```

As a result, I got the `call.yml` script which looked like this:

```yml
---
- send:
    data:
      event: connected
      protocol: Call
      version: 0.2.0
- sleep:
    time: 0.022
- send:
    data:
      event: start
      sequenceNumber: '1'
      start:
        accountSid: AC41149b360
        streamSid: MZ50f966ef8a
        callSid: CAa910aa778
        tracks:
        - inbound
        mediaFormat:
          encoding: audio/x-mulaw
          sampleRate: 8000
          channels: 1
        customParameters:
          from: client:Anonymous
          to: ''
      streamSid: MZ50f966ef8a
- sleep:
    time: 0.021
# ...
- send:
    data:
      event: media
      sequenceNumber: '139'
      media:
        track: inbound
        chunk: '138'
        timestamp: '2891'
        payload: "//9+/v3+/f1+fv5+fX5+/3+e37+//7+/w=="
      streamSid: MZ50f966ef8a
# ... thousands lines of code
- send:
    data:
      event: stop
      sequenceNumber: '1107'
      streamSid: MZ50f966ef8a
      stop:
        accountSid: AC41149b360
        callSid: CAa910aa778
- sleep:
    time: 0.02
```

To emulate a phone call, I just ran the following command without leaving my IDE:

```sh
wsdirector fixtures/wsdirector/call.yml ws://localhost:9010/twilio
```

This saved me an incredible amount of time, and as such, I could actually focus on the real problem. However, I can't say this application of
Websocket Director was crucial to the application's development. Rather, it was about pure developer experience.

Let's move on to the next story, in which WebSocket Director became a part of the application lifecycle.

## WS Director vs. system testing

*Read also Evil Martians' article*: https://evilmartians.com/chronicles/system-of-a-test-setting-up-end-to-end-rails-testing

Writing integration tests for WebSockets-driven functionality is not rocket science. We can use our beloved Capybara and Rails system tests to emulate user interactions with our application via a web browser. However, not every interaction can be reproduced in a browser: for example, if we have a mobile application in our "play".

So, here is a user scenario I had to transform into a system test:

- User C (_driver_) uses a mobile application, which tracks their GPS location and sends it to the _tracker server_.
- User A (_admin_) opens a product dashboard, which contains the active drivers list with their real-time locations.
- We want to ensure that whenever a driver's mobile application sends location updates, an admin can see them immediately.

How to write such a test? We can't run a mobile app simulator and control it with Capybara. You might think of triggering broadcasts manually in a test, e.g., `ActionCable.server.broadcast "locations", data`. Yeah, that could be a workaround (although, in my opinion, calling _internal_ code makes our test escape from the black box). In our case, we used a standalone WebSocket server, which was created specifically for this task and had no public APIs to perform broadcasts from the outside. The only way to emulate the GPS update broadcast was to communicate over a WebSocket connection.

The tracking server was written in Elixir and used [Phoenix Channels][ph-channels] under the hood. Did I mention that WS Director is protocol-agnostic? Although it was built with Action Cable in mind, it could be used to interact with any WebSockets server by using the _primitive_ actions: `send`, `receive`, `sleep`, etc.

To emulate a mobile application activity, we created the following script:

```yml
---
- send:
    data:
      topic: tenant:<%= ENV.fetch('TENANT', 'rspec') %>
      event: phx_join
      payload: {}
      ref: '1'
- receive:
    data:
      topic: tenant:<%= ENV.fetch('TENANT', 'rspec') %>
      ref: '1'
      payload:
        status: ok
        response: {}
      event: phx_reply
- send:
    data:
      topic: tenant:<%= ENV.fetch('TENANT', 'rspec') %>
      event: update_location
      payload:
        position:
          latitude: <%= ENV.fetch('LAT', 32.84019785216758).to_f %>
          longitude: <%= ENV.fetch('LON', -97.06401083105213).to_f %>
        technicianId: <%= ENV['ID'] %>
      ref: '2'
```

**NOTE:** The tracking app was built with an older version of Phoenix and used an outdated version of the Channels protocol.

That worked perfectly in development. But what about tests? How to invoke WebSocket Director from a Rails system test?

One option could be to "sell out" and simply call the `wsdirector` CLI. However, since the tool is written in Ruby, why not use it directly? The final test example code looked like this:

```ruby
it "sees location updates in real-time" do
  within "#tech-#{tech.id} .location" do
    expect(page).to have_text "Bronx, NY"
  end

  run_websocket_scenario(
    "tracker/update_location.yml",
    token: jwt_token,
    env: {
      "ID" => tech.id,
      "LAT" => 32.9846003797191,
      "LON" => -97.0647746830336
    }
  )

  within "#tech-#{tech.id} .location" do
    expect(page).to have_text "Dallas, TX"
  end
end
```

What's hidden inside the `#run_websocket_scenario` method? Well, I'd prefer not to show the original contents—they didn't look great. And the reason for that is because the gem wasn't initially designed for programmatic usage; it was just a CLI tool. For example, it relied on a global configuration object, and the only way to add dynamic values was by using environment variables.

Using WS Director in commercial projects, I collected enough feedback to realize that, first of all, the gem is more generic than just a dev tool for Action Cable or AnyCable developers; and second, some refactoring is needed. This revelation led to the first major release of WS Director. Allow me to introduce WebSocket Director 1.0!

## Introducing WS Director 1.0

I doubt many readers were familiar with WebSocket Director prior to this post, so there's no need to describe the actual changes (but you can check the [release notes][], if you're so inclined). Instead, I'd rather show off what you can do with WS Director 1.0.

First, it now provides a much better experience when running from Ruby. For instance, the source code of the `run_websocket_scenario` now looks like this:

```ruby
def run_websocket_scenario(path, token:, url: Rails.configuration.tracker_url, **options)
  url = "#{url}?token=#{token}"
  scenario = Rails.root.join "spec" / "fixtures" / "wsdirector" / path

  WSDirector.run(scenario, url:, **options)
end

# Usage in tests
run_websocket_scenario(
  "tracker/update_location.yml",
  token: jwt_token,
  locals: {
    id: tech.id,
    tenant: tech.tenant,
    lat: 32.9846003797191,
    lon: -97.0647746830336
  }
)
```

Note that we replaced the env vars with `locals`. That's because you can now pass local variables directly to scenarios! Here's our updated scenario:

```yml
- client:
    protocol: phoenix
    actions:
      - join:
          topic: tenant:<%= tenant %>
      - send:
          topic: tenant:<%= tenant %>
          event: update_location
          data:
            position:
              latitude: <%= lat %>
              longitude: <%= lon %>
            technicianId: <%= id %>
```

Oh, did you notice that the script itself has become much more concise? That's because WS Director now supports the Elixir Channels protocol out-of-the-box.

Speaking of protocols, it's also possible now to load custom protocols dynamically when using the `wsdirector` CLI:

```sh
wsdirector -r ./my_protocol.rb -u localhost:3030/ws -i '[{"client":{"protocol":"MyProtocol"}}]'
```

And last but not least, I'd like to talk about one more feature: the verbose output mode. And no words needed, just watch the video:

---

Developer tools make our lives better, even if you've made one just to satisfy your own particular needs. But even if you've built something tailored specifically for what you want, it doesn't mean it won't be up there one day, standing on the stage in front of the bright lights for all the world to see...

And, by the way, if your project has any backstage issues to be solved, of if it needs a hand getting ready for a big stage debut, [give Evil Martians a buzz!](https://evilmartians.com/contact-us)

[anycable]: https://anycable.io
[wsdirector]: https://evilmartians.com/opensource/wsdirector
[action-cable-protocol]: https://docs.anycable.io/misc/action_cable_protocol
[pgbench]: https://www.postgresql.org/docs/current/pgbench.html
[k6]: https://k6.io
[websocket-bench]: https://github.com/anycable/websocket-bench
[xk6-cable]: https://evilmartians.com/opensource/xk6-cable
[acli]: https://evilmartians.com/opensource/acli
[twilio]: https://www.twilio.com
[twilio-voice]: https://www.twilio.com/docs/voice
[ph-channels]: https://hexdocs.pm/phoenix/channels.html
[release notes]: https://github.com/palkan/wsdirector/releases/tag/v1.0.0

---

This article was written by [Evil Martians](https://evilmartians.com), a product development consultancy trusted by teams from early-stage startups to public companies. If you're working through a hard problem, [we can help](https://evilmartians.com/contact-us)
