Ruby on Whales: Dockerizing Ruby and Rails development

March 15, 2022

Topics

Vladimir Dementyev
Principal Backend Engineer

Translations

This post introduces the Docker configuration I use for developing my Ruby on Rails projects. This configuration came out of—and then further evolved—during ~~production~~ development at Evil Martians. Read on to learn all the details, and feel free to use it, share it, and enjoy!

Notice: This article is regularly updated with the best and latest recommendations; for details, take a look at the Changelog.

So, where to start? This has been a pretty long journey: back in the day, I used to develop using Vagrant, but its VMs were a bit too heavy for my 4GB RAM laptop. In 2017, I decided to make the switch to containers, and this was how I first began using Docker. But don’t get the impression that this was an instant fix! I was in search of a configuration that was perfect for myself, my team, and well, everyone else. And something which was just good enough would not cut it. It took quite some time to develop a standard approach (as more formerly enshrined with the first release of this article in 2019). Since that first iteration of this post revealed my secret to the world, many Rails teams and devs have adopted my technique, and actually, they’ve helped to contribute and improve it!

With that out of the way, let me just go ahead and present the config itself. Along the way, I’ll explain almost every line (because we’ve all had enough of those cryptic tutorials that just assume you know stuff).

The source code can be found in the evilmartians/ruby-on-whales repository on GitHub.

Before we get on with it, let’s note that we’ll be using up-to-date software versions for this example: Docker Desktop 20.10+ and Docker Compose v2, Ruby 3.1.0, PostgreSQL 14, etc.

The bulk of the post consists mostly of annotated code and configuration examples, structured as follows:

The basics: Dockerfile and docker-compose.yml
Introducing Dip
(Micro-)services vs Docker for development
From development to production
Introducing the Ruby on Whales interactive generator

Basic Docker configuration

`Dockerfile`

The Dockerfile defines our Ruby application’s environment. This environment is where we’ll run servers, access the console (rails c), perform tests, do Rake tasks, and otherwise interact with our code in any way as developers:

# syntax=docker/dockerfile:1

ARG RUBY_VERSION
ARG DISTRO_NAME=bullseye

FROM ruby:$RUBY_VERSION-slim-$DISTRO_NAME

ARG DISTRO_NAME

# Common dependencies
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  rm -f /etc/apt/apt.conf.d/docker-clean; \
  echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache; \
  apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    gnupg2 \
    curl \
    less \
    git

# Install PostgreSQL dependencies
ARG PG_MAJOR
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
    gpg --dearmor -o /usr/share/keyrings/postgres-archive-keyring.gpg \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/postgres-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt/" \
    $DISTRO_NAME-pgdg main $PG_MAJOR | tee /etc/apt/sources.list.d/postgres.list > /dev/null
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    libpq-dev \
    postgresql-client-$PG_MAJOR

# Install NodeJS and Yarn
ARG NODE_MAJOR
ARG YARN_VERSION=latest
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    --mount=type=tmpfs,target=/var/log \
    apt-get update && \
    apt-get install -y curl software-properties-common && \
    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
    echo "deb https://deb.nodesource.com/node_${NODE_MAJOR}.x $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/nodesource.list && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends nodejs
RUN npm install -g yarn@$YARN_VERSION

# Application dependencies
# We use an external Aptfile for this, stay tuned
COPY Aptfile /tmp/Aptfile
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    $(grep -Ev '^\s*#' /tmp/Aptfile | xargs)

# Configure bundler
ENV LANG=C.UTF-8 \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3

# Store Bundler settings in the project's root
ENV BUNDLE_APP_CONFIG=.bundle

# Uncomment this line if you want to run binstubs without prefixing with `bin/` or `bundle exec`
# ENV PATH /app/bin:$PATH

# Upgrade RubyGems and install the latest Bundler version
RUN gem update --system && \
    gem install bundler

# Create a directory for the app code
RUN mkdir -p /app
WORKDIR /app

# Document that we're going to expose port 3000
EXPOSE 3000
# Use Bash as the default command
CMD ["/bin/bash"]

This configuration only contains the essentials, and so it can be used as a starting point. Let me illustrate what we’re are doing here a bit further. The first three lines might look a bit strange:

ARG RUBY_VERSION
ARG DISTRO_NAME=bullseye
FROM ruby:$RUBY_VERSION-slim-$DISTRO_NAME

Why not just use FROM ruby:3.1.0, or whatever is the stable Ruby version du jour? Well, we’re going this route because we want to make our environment configurable from the outside using Dockerfile as a sort of a template:

The exact versions of the runtime dependencies are specified in the docker-compose.yml (see below 👇).
The list of apt-installable dependencies is stored in a separate file (also, see below 👇👇).

Additionally, we parameterize the Debian release (bullseye by default) to make sure we’re adding the correct sources for our other dependencies (such as PostgreSQL).

Alright, now, note that we declare the argument once again after the FROM statement:

FROM ruby:$RUBY_VERSION-slim-$DISTRO_NAME
ARG DISTRO_NAME

That’s the tricky part of how Dockerfiles work: the args are reset after the FROM statement. For more details, check out this issue.

Moving on, the rest of the file contains the actual build steps. First, we’ll need to manually install some common system dependencies (Git, cURL, etc.), as we’re using the slim base Docker image to reduce the size:

# Common dependencies
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    gnupg2 \
    curl \
    less \
    git

We’ll explain all the details of installing system dependencies below, alongside the application-specific dependencies.

Installing PostgreSQL and NodeJS via apt requires adding their deb package repos to the sources list.

Here’s PostgreSQL (based on the official documentation):

ARG PG_MAJOR
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
    gpg --dearmor -o /usr/share/keyrings/postgres-archive-keyring.gpg \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/postgres-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt/" \
    $DISTRO_NAME-pgdg main $PG_MAJOR | tee /etc/apt/sources.list.d/postgres.list > /dev/null
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    libpq-dev \
    postgresql-client-$PG_MAJOR

Since we aren’t expecting anyone to use this Dockerfile without Docker Compose, we don’t provide a default value for the PG_MAJOR argument (the same applies to NODE_MAJOR below, and YARN_VERSION further below).

Also, notice that in the code above that the DISTRO_NAME argument which we defined at the very beginning of the file comes back into play.

And, we repeat our apt-get ... apt-get clean spell again: we want to make sure all the major pieces of our environment are built in an isolated way (this will help us to better utilize Docker cache layers when performing upgrades).

For NodeJS (from the NodeSource repo):

ARG NODE_MAJOR
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    --mount=type=tmpfs,target=/var/log \
    apt-get update && \
    apt-get install -y curl software-properties-common && \
    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
    echo "deb https://deb.nodesource.com/node_${NODE_MAJOR}.x $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/nodesource.list && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends nodejs

Then, we install Yarn via NPM:

ARG YARN_VERSION=latest
RUN npm install -g yarn@$YARN_VERSION

So, why are we adding NodeJS and Yarn in the first place? Although Rails 7 allows you to go Node-less via import maps or precompiled binaries (like tailwindcss-rails), these additions increase the chances of supporting legacy pipelines or adding modern Webpacker alternatives.

Now it’s time to install the application-specific dependencies:

COPY Aptfile /tmp/Aptfile
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    $(grep -Ev '^\s*#' /tmp/Aptfile | xargs)

Let’s talk about that Aptfile trick a bit:

COPY Aptfile /tmp/Aptfile
RUN apt-get install \
    $(grep -Ev '^\s*#' /tmp/Aptfile | xargs) \

I borrowed this idea from heroku-buildpack-apt, which allows for installing additional packages on Heroku. If you’re using this buildpack, you can even re-use the same Aptfile for both the local and the production environment.

Our default Aptfile contains only a single package (we’ll use Vim to edit the Rails Credentials):

vim

In one of the previous projects I worked on, we generated PDFs using LaTeX and TexLive. In a case like that, our Aptfile might look like this:

vim
# TeX packages
texlive
texlive-latex-recommended
texlive-fonts-recommended
texlive-lang-cyrillic

By doing this we can keep task-specific dependencies in a separate file, thus making our Dockerfile more universal.

With regards to DEBIAN_FRONTEND=noninteractive, I kindly ask you to take a look at this answer on Ask Ubuntu.

The --no-install-recommends option helps save some space (and makes our image smaller) by disabling the installation of recommended packages. You can see more about saving disk space here.

That first (fairly cryptic) part of every RUN statement that installs packages also serves the same purpose: it moves out the local repository of retrieved package files into a cache that will be preserved between builds. We need this magic to be in every RUN statement that installs packages to make sure this particular Docker layer doesn’t contain any garbage. It also greatly speeds up image build!

The final part of the Dockerfile is mostly devoted to Bundler:

# Configure bundler
ENV LANG=C.UTF-8 \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3 \

# Store Bundler settings in the project's root
ENV BUNDLE_APP_CONFIG=.bundle

# Uncomment this line if you want to run binstubs without prefixing with `bin/` or `bundle exec`
# ENV PATH /app/bin:$PATH

# Upgrade RubyGems and install the latest Bundler version
RUN gem update --system && \
    gem install bundler

Using LANG=C.UTF-8 sets the default locale to UTF-8. This is an emotional setting, as otherwise, Ruby would use US-ASCII for strings—and that’d mean waving goodbye to those sweet, sweet emojis! 👋

Setting BUNDLE_APP_CONFIG is required if you’ll use the <root>/.bundle folder to store project-specicic Bundler settings (like credentials for private gems). The default Ruby image defines this variable so Bundler doesn’t fall back to the local config.

Optionally, you can add your <root>/bin folder to the PATH in order to run commands without bundle exec. We don’t do this by default, because it could break in a multi-project environment (for instance, when you have local gems or engines in your Rails app).

Previously, we also had to specify the Bundler version (taking advantage of some hacks to make sure it’s picked up by the system). Luckily, since Bundler 2.3.0, we no longer need to manually install the version defined in the Gemfile.lock (BUNDLED_WITH). Instead, to avoid conflicts, Bundler does this for us.

`compose.yml`

Docker Compose is a tool we can use to orchestrate our containerized environment. It allows us to link containers to each other, and to define persistent volumes and services.

Below is the compose file for developing a typical Rails application with PostgreSQL as the database, and with Sidekiq as the background job processor:

x-app: &app
  build:
    context: .
    args:
      RUBY_VERSION: '3.2.2'
      PG_MAJOR: '15'
      NODE_MAJOR: '18'
  image: example-dev:1.0.0
  environment: &env
    NODE_ENV: ${NODE_ENV:-development}
    RAILS_ENV: ${RAILS_ENV:-development}
  tmpfs:
    - /tmp
    - /app/tmp/pids

x-backend: &backend
  <<: *app
  stdin_open: true
  tty: true
  volumes:
    - ..:/app:cached
    - bundle:/usr/local/bundle
    - rails_cache:/app/tmp/cache
    - node_modules:/app/node_modules
    - packs:/app/public/packs
    - packs-test:/app/public/packs-test
    - history:/usr/local/hist
    - ./.psqlrc:/root/.psqlrc:ro
    - ./.bashrc:/root/.bashrc:ro
  environment: &backend_environment
    <<: *env
    REDIS_URL: redis://redis:6379/
    DATABASE_URL: postgres://postgres:postgres@postgres:5432
    WEBPACKER_DEV_SERVER_HOST: webpacker
    MALLOC_ARENA_MAX: 2
    WEB_CONCURRENCY: ${WEB_CONCURRENCY:-1}
    BOOTSNAP_CACHE_DIR: /usr/local/bundle/_bootsnap
    XDG_DATA_HOME: /app/tmp/caches
    YARN_CACHE_FOLDER: /app/node_modules/.yarn-cache
    HISTFILE: /usr/local/hist/.bash_history
    PSQL_HISTFILE: /usr/local/hist/.psql_history
    IRB_HISTFILE: /usr/local/hist/.irb_history
    EDITOR: vi
  depends_on: &backend_depends_on
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy

services:
  rails:
    <<: *backend
    command: bundle exec rails

  web:
    <<: *backend
    command: bundle exec rails server -b 0.0.0.0
    ports:
      - '3000:3000'
    depends_on:
      webpacker:
        condition: service_started
      sidekiq:
        condition: service_started

  sidekiq:
    <<: *backend
    command: bundle exec sidekiq -C config/sidekiq.yml

  postgres:
    image: postgres:14
    volumes:
      - .psqlrc:/root/.psqlrc:ro
      - postgres:/var/lib/postgresql/data
      - history:/user/local/hist
    environment:
      PSQL_HISTFILE: /user/local/hist/.psql_history
      POSTGRES_PASSWORD: postgres
    ports:
      - 5432
    healthcheck:
      test: pg_isready -U postgres -h 127.0.0.1
      interval: 5s

  redis:
    image: redis:6.2-alpine
    volumes:
      - redis:/data
    ports:
      - 6379
    healthcheck:
      test: redis-cli ping
      interval: 1s
      timeout: 3s
      retries: 30

  webpacker:
    <<: *app
    command: bundle exec ./bin/webpack-dev-server
    ports:
      - '3035:3035'
    volumes:
      - ..:/app:cached
      - bundle:/usr/local/bundle
      - node_modules:/app/node_modules
      - packs:/app/public/packs
      - packs-test:/app/public/packs-test
    environment:
      <<: *env
      WEBPACKER_DEV_SERVER_HOST: 0.0.0.0
      YARN_CACHE_FOLDER: /app/node_modules/.yarn-cache

volumes:
  bundle:
  node_modules:
  history:
  rails_cache:
  postgres:
  redis:
  packs:
  packs-test:

We define six services and two extension fields (x-app and x-backend). Extension fields allow us to define common parts of the configuration. We can attach YAML anchors to them, and later, embed anywhere in the file.

NOTE: In the end, we don’t use Docker Compose or execute the docker compose up command in order to run our application. Instead, we use Dip (see below), and thus, the compose.yml file only acts as a services registry. Another important thing to note is that we put the compose.yml file into the .dockerdev/ folder. This is why we mount the source code as ..:/app and not .:/app. Please, keep this in mind if you’re considering using this configuration without Dip (which is not recommended).

On that note, let’s go ahead and take a thorough look at each service.

`x-app`

The main purpose of this extension is to provide all the required information to build our application container (as defined in the Dockerfile above):

x-app: &app
  build:
    context: .
    args:
      RUBY_VERSION: '3.2.2'
      PG_MAJOR: '15'
      NODE_MAJOR: '18'

What is the context? The context directory defines the build context for Docker. This is something like a working directory for the build process—for example, when we execute the COPY command. As this directory is packaged and sent to the Docker daemon every time an image is built, it’s better to keep it as small as possible. We’re good here, since our context is just the .dockerdev folder.

And, as we mentioned earlier, we’ll specify the exact version of our dependencies using the args as declared in the Dockerfile.

It’s also a good idea to pay attention to the way we tag images:

image: example-dev:1.0.0

One of the benefits of using Docker for development is the ability to automatically synchronize configuration changes across the team. This means the only time you need to upgrade the local image version is when you make changes to it (or to the arguments or files it relies on). Using example-dev:latest is like shooting yourself in the foot.

Keeping an image version also helps work with two different environments without any additional hassle. For example, when working on a long-standing “chore/upgrade-to-ruby-3” branch, you can easily switch to master and use the older image with the older version of Ruby: no need to rebuild anything.

Rule of thumb: Increase the version number in the image tag every time you change Dockerfile or its arguments (upgrading dependencies, etc.)

Next, we add some common environment variables (those shared by multiple services, e.g., Rails and Webpacker):

environment: &env
  NODE_ENV: ${NODE_ENV:-development}
  RAILS_ENV: ${RAILS_ENV:-development}

There are several things going on here, but I’d like to focus on just one: the X=${X:-smth} syntax. This could be translated as “For X variable within the container, if present, use the host machine’s X env variable, otherwise, use another value”. Thus, we make it possible to run a service in a different environment specified along with a command, e.g., RAILS_ENV=test docker-compose up rails.

Note that we’re using a dictionary value (NODE_ENV: xxx) and not a list value (- NODE_ENV=xxx) for the environment field. This allows us to re-use the common settings (see below).

We also tell Docker to use tmpfs for the /tmp folder within a container—and also for the tmp/pids folder of our application. This way, we ensure that no server.pid survives a container exit (say goodbye to any “A server is already running” errors):

tmpfs:
  - /tmp
  - /app/tmp/pids

`x-backend`

Alright, so now, we’ve finally reached the most interesting part of this post.

This service defines the shared behavior of all Ruby services.

Let’s talk about the volumes first:

x-backend: &backend
  <<: *app
  stdin_open: true
  tty: true
  volumes:
    - ..:/app:cached
    - rails_cache:/app/tmp/cache
    - bundle:/usr/local/bundle
    - history:/usr/local/hist
    - node_modules:/app/node_modules
    - packs:/app/public/packs
    - packs-test:/app/public/packs-test
    - ./.psqlrc:/root/.psqlrc:ro
    - ./.bashrc:/root/.bashrc:ro
    - ./.pryrc:/root/.pryrc:ro
  environment: &backend_environment
    <<: *env
    REDIS_URL: redis://redis:6379/
    DATABASE_URL: postgres://postgres:postgres@postgres:5432
    WEBPACKER_DEV_SERVER_HOST: webpacker
    MALLOC_ARENA_MAX: 2
    WEB_CONCURRENCY: ${WEB_CONCURRENCY:-1}
    BOOTSNAP_CACHE_DIR: /usr/local/bundle/_bootsnap
    XDG_DATA_HOME: /app/tmp/caches
    YARN_CACHE_FOLDER: /app/node_modules/.yarn-cache
    HISTFILE: /usr/local/hist/.bash_history
    PSQL_HISTFILE: /usr/local/hist/.psql_history
    IRB_HISTFILE: /usr/local/hist/.irb_history
    EDITOR: vi
  depends_on: &backend_depends_on
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy
volumes:
  - ..:/app:cached
  - bundle:/usr/local/bundle
  - rails_cache:/app/tmp/cache
  - node_modules:/app/node_modules
  - packs:/app/public/packs
  - packs-test:/app/public/packs-test
  - history:/usr/local/hist
  - ./.psqlrc:/root/.psqlrc:ro
  - ./.bashrc:/root/.bashrc:ro

The first item in the volumes list mounts the project directory to the /app folder within a container using the cached strategy. This cached modifier was the key to efficient Docker development on macOS.

Wait, was?

Yeah. Was. That’s because since the release of gRPC FUSE synchronization, it’s no longer needed. Still, I decided to keep it for a while, for two reasons: first, some of your teammates may still be using older Docker desktop versions, and second, I ran some benchmarks and found that using older osxfs file sharing could have better performance (but only when using :cached). So, even on modern versions of Docker, it could make sense to uncheck the “Use gRPC FUSE for file sharing” option inside the preferences menu.

The next line tells our container to use a volume named bundle to store the contents of /usr/local/bundle (this is where gems are stored by default). By doing this, we persist our gem data across runs: all the volumes defined in compose.yml will stay put until we run compose down --volumes.

The following lines have also been dutifully placed in order to nullify the “Docker is slow on Mac” curse. We put all the generated files into Docker volumes to avoid any heavy disk operations on the host machine:

- rails_cache:/app/tmp/cache
- node_modules:/app/node_modules
- packs:/app/public/packs
- packs-test:/app/public/packs-test

To give Docker a suitably fast speed on macOS, follow these two rules: use :cached to mount source files (if not using gRPC FUSE), and use volumes for generated content (assets, bundle, etc.).

NOTE: If you’re using Sprockets (or Propshaft), don’t forget to add a dedicated volume to store the assets (assets:/app/public/assets). For tailwindcss-rails, add something like assets_builds:/app/assets/builds.

We’ll then mount different command line tools configuration files and a volume to persist their history:

- history:/usr/local/hist
- ./.psqlrc:/root/.psqlrc:ro
- ./.bashrc:/root/.bashrc:ro

Oh, and why is psql in the Ruby container? That’s because it’s used internally when you run rails dbconsole.

Pressing onward, our .psqlrc file contains the following trick which makes it possible to specify the path to the history file via the env variable—thus allowing us to specify the path to the history file via the PSQL_HISTFILE env variable, or otherwise, fall back to the default $HOME/.psql_history:

\set HISTFILE `[[ -z $PSQL_HISTFILE ]] && echo $HOME/.psql_history || echo $PSQL_HISTFILE`

The .bashrc file allows us to add terminal customizations within a container:

alias be="bundle exec"

Alright, let’s talk about the environment variables:

environment: &backend_environment
  <<: *env
  # ----
  # Service discovery
  # ----
  REDIS_URL: redis://redis:6379/
  DATABASE_URL: postgres://postgres:postgres@postgres:5432
  WEBPACKER_DEV_SERVER_HOST: webpacker
  # ----
  # Application configuration
  # ----
  MALLOC_ARENA_MAX: 2
  WEB_CONCURRENCY: ${WEB_CONCURRENCY:-1}
  # -----
  # Caches
  # -----
  BOOTSNAP_CACHE_DIR: /usr/local/bundle/_bootsnap
  # This env variable is used by some tools (e.g., RuboCop) to store caches
  XDG_DATA_HOME: /app/tmp/cache
  # Puts the Yarn cache into a mounted volume for speed
  YARN_CACHE_FOLDER: /app/node_modules/.yarn-cache
  # ----
  # Dev tools
  # ----
  HISTFILE: /usr/local/hist/.bash_history
  PSQL_HISTFILE: /usr/local/hist/.psql_history
  IRB_HISTFILE: /usr/local/hist/.irb_history
  EDITOR: vi

First of all, we “inherit” variables from the common environment variables (<<: *env).

The first group of variables (DATABASE_URL, REDIS_URL, and WEBPACKER_DEV_SERVER_HOST) connect our Ruby application to other services.

The DATABASE_URL and WEBPACKER_DEV_SERVER_HOST variables are supported by Rails (ActiveRecord and Webpacker respectively) out of the box. Some libraries (Sidekiq) also support REDIS_URL, but not all of them: for instance, Action Cable must be explicitly configured.

The second group contains some application-wide settings. For example, we define MALLOC_ARENA_MAX and WEB_CONCURRENCY to help us keep Ruby memory handling in check.

Also, we have the variables responsible for storing caches in Docker volumes (BOOTSNAP_CACHE_DIR, XDG_DATA_HOME, YARN_CACHE_FOLDER).

We use bootsnap to speed up application load time. We store its cache in the same volume as the Bundler data. This is because this cache mostly contains the gem data, and we want to make sure the cache is reset every time we drop the Bundler volume (for instance, during a Ruby version upgrade).

The final group of variables aim to improve the developer experience. HISTFILE: /usr/local/hist/.bash_history is the most significant here: it tells Bash to store its history in the specified location, thus making it persistent. The same goes for PSQL_HISTFILE and IRB_HISTFILE.

NOTE: You need to configure IRB to store history in the specified location. To do that, drop these lines into your .irbrc file:

IRB.conf[:HISTORY_FILE] = ENV["IRB_HISTFILE"] if ENV["IRB_HISTFILE"]

Finally, EDITOR: vi is used, for example, by the rails credentials:edit command to manage credentials files.

And with that, the only lines in this service we’ve yet to cover are:

stdin_open: true
tty: true

These lines make this service interactive, that is, they provide a TTY. We need this, for example, to run the Rails console or Bash within a container.

This is the same as running a Docker container with the -it option.

`rails`

The rails server is our default backend service. The only thing it overrides is the command to execute:

rails:
  <<: *backend
  command: bundle exec rails

This service is meant for executing all the commands needed in development (rails db:migrate, rspec, etc.).

`web`

The web service is meant for launching a web server. It defines the exposed ports and the required dependencies to run the app itself.

`webpacker`

The only thing I want to mention here is the WEBPACKER_DEV_SERVER_HOST: 0.0.0.0 setting: it makes the Webpack dev server accessible from the outside (it runs on localhost by default).

Health checks

When running common Rails commands such as db:migrate, we want to ensure that the DB is up and ready to accept connections. How can we tell Docker Compose to wait until a dependent service is ready? We can use health checks!

You’ve probably noticed that our depends_on definition isn’t just a list of services:

backend:
  # ...
  depends_on:
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy

postgres:
  # ...
  healthcheck:
    test: pg_isready -U postgres -h 127.0.0.1
    interval: 5s

redis:
  # ...
  healthcheck:
    test: redis-cli ping
    interval: 1s
    timeout: 3s
    retries: 30

Introducing Dip

If you still think that Docker Compose way is too complicated, there’s a tool called Dip (developed by one of my colleages at Evil Martians) which aims to make the developer experience even smoother.

Dip is a thin wrapper over docker compose, which provides a switch from infrastructure-oriented flow to development-oriented one. The key benefits of using Dip are as follows:

The ability to define application-specific interactive commands and sub-commands.
The dip provision flow to quickly set up a development environment from scratch.
Support for multiple compose.yml files (including OS-specific configurations).

With Dip in place, to start working on the app locally, you just need to execute a few commands:

# Builds a Docker image if none, runs additional commands
$ dip provision
# Runs a Rails server with the defined dependencies
$ dip rails s
=> Booting Puma
=> Rails 7.0.2.2 application starting in development
=> Run `bin/rails server --help` for more startup options
[1] Puma starting in cluster mode...
...
[1] - Worker 0 (PID: 9) booted in 0.0s, phase: 0

Here is our typical dip.yml file:

version: '7.1'

# Define default environment variables to pass
# to Docker Compose
environment:
  RAILS_ENV: development

compose:
  files:
    - .dockerdev/compose.yml
  project_name: example_demo

interaction:
  # This command spins up a Rails container with the required dependencies (such as databases),
  # and opens a terminal within it.
  runner:
    description: Open a Bash shell within a Rails container (with dependencies up)
    service: rails
    command: /bin/bash

  # Run a Rails container without any dependent services (useful for non-Rails scripts)
  bash:
    description: Run an arbitrary script within a container (or open a shell without deps)
    service: rails
    command: /bin/bash
    compose_run_options: [ no-deps ]

  # A shortcut to run Bundler commands
  bundle:
    description: Run Bundler commands
    service: rails
    command: bundle
    compose_run_options: [ no-deps ]

  # A shortcut to run RSpec (which overrides the RAILS_ENV)
  rspec:
    description: Run RSpec commands
    service: rails
    environment:
      RAILS_ENV: test
    command: bundle exec rspec

  rails:
    description: Run Rails commands
    service: rails
    command: bundle exec rails
    subcommands:
      s:
        description: Run Rails server at http://localhost:3000
        service: web
        compose:
          run_options: [service-ports, use-aliases]

  yarn:
    description: Run Yarn commands
    service: rails
    command: yarn
    compose_run_options: [ no-deps ]

  psql:
    description: Run Postgres psql console
    service: postgres
    default_args: anycasts_dev
    command: psql -h postgres -U postgres

  'redis-cli':
    description: Run Redis console
    service: redis
    command: redis-cli -h redis

provision:
  - dip compose down --volumes
  - dip compose up -d postgres redis
  - dip bash -c bin/setup

Let me explain some bits of this in further detail.

First, the compose section:

compose:
  files:
    - .dockerdev/compose.yml
  project_name: example_demo

Here we should specify the path to our Compose configuration (.dockerdev/compose.yml). Accordingly, we can run dip from the project root, and the correct configuration will be picked up.

The project_name is important: if we don’t specify it, the folder containing the compose.yml file would be used (“dockerdev”), which could lead to collisions between different projects.

The rails command is also worth some additional attention:

rails:
  description: Run Rails commands
  service: rails
  command: bundle exec rails
  subcommands:
    s:
      description: Run Rails server at http://localhost:3000
      service: web
      compose:
        run_options: [service-ports, use-aliases]

By default, the dip rails command would call bundle exec rails within a Rails container. However, we use the subcommand feature of Dip here to treat dip rails s differently:

We use the web service, not rails (so, the deps are up).
We expose the service ports (3000 in our case).
We also enable network aliases, so other services can access this container via the web hostname.

Under the hood, this will result in the following Docker Compose command:

docker compose run --rm --service-ports --use-aliases web

Note that it uses run, and not up. This difference makes our server terminal-accessible. For example, this means that we can attach a debugger and use it without any problems (with the up command the terminal is non-interactive).

Interactive provisioning

For most applications, building an image and setting up a database is not enough to start developing: beyond this, some kind of secrets, or credentials, or .env files are required. Here, we’ve managed to use Dip to help new engineers quickly assemble all these wayfallen parts by providing an interactive provision experience.

Let’s consider, for example, that we need to put a .env.development.local file with some secret info and also configure RubyGems to download packages from a private registry (say, Sidekiq Pro). For this, I’ll write the following provision script:

# The command is extracted, so we can use it alone
configure_bundler:
  command: |
    (test -f .bundle/config && cat .bundle/config | \
      grep BUNDLE_ENTERPRISE__CONTRIBSYS__COM > /dev/null) ||
    \
      (echo "Sidekiq ent credentials: "; read -r creds; dip bundle config --local enterprise.contribsys.com $creds)

provision:
  - (test -f .env.development.local) || (echo "\n\n ⚠️  .env.development.local file is missing\n\n"; exit 1)
  - dip compose down --volumes
  - dip configure_bundler
  - (test -f config/database.yml) || (cp .dockerdev/database.yml.example config/database.yml)
  - dip compose up -d postgres redis
  - dip bash -c bin/setup

Below you can see a demonstration of this command running in action:

An interactive Dip provisioning example

Services vs Docker for development

One more use case for standardizing the development setup is to make it possible to run multiple independent services locally. Let me quickly demonstrate how we do this with Dip. First, you need to dockerize each application (following this post). After that, we need to connect the apps to each other. How can we do this? With the help of Docker Compose external networks.

We add the following line to the dip.yml for each app:

# ...
provision:
  # Make sure the named network exists
  - docker network inspect my_project > /dev/null 2>&1 || \
    docker network create my_project
# ...

Finally, we attach services to this network via aliases in the compose.yml files:

# service A: compose.yml
service:
  ruby:
    # ...
    networks:
      default:
      project:
        aliases:
          - project-a

networks:
  project:
    external:
      name: my_project

# service B: compose.yml
service:
  web:
    # ...
    environment:
      # We can access the service A via its alias defined for the external network
      SERVICE_URL: http://project-a:3000

networks:
  project:
    external:
      name: my_project

From development to production

So, here’s one of the most popular questions we’ve been asked since launching the first version of this article: how to go live with Docker? To answer this, we’d need to write a entirely new article… and we will 😉.

For now, let me give a sneak preview of how can we extend the current development setup to cover the production environment as well.

First of all, we’re not going to talk about a Docker Compose-based deployment, so compose.yml is out. All we need is to update our Docker image to reflect the difference between development and production:

For security reasons, we should execute the code on behalf of the regular, non-root user.
We should keep all the required dependencies and artifacts within the image itself; we cannot use volumes (the image should be self-contained).
We should keep and copy the source code into a container.
The resulting image should be as slim as possible.

To achieve this, we’ll refactor our existing Dockerfile to define multiple stages (and to support multi-stage builds). Below is the annotated example:

# syntax=docker/dockerfile:1

ARG RUBY_VERSION
ARG DISTRO_NAME=bullseye

# Here we add the the name of the stage ("base")
FROM ruby:$RUBY_VERSION-slim-$DISTRO_NAME AS base

ARG PG_MAJOR
ARG NODE_MAJOR
ARG YARN_VERSION

# Common dependencies
# ...
# The following lines are exactly the same as before
# ...
# ...
WORKDIR /app

EXPOSE 3000
CMD ["/bin/bash"]

# Then, we define the "development" stage from the base one
FROM base AS development

ENV RAILS_ENV=development

# The major difference from the base image is that we may have development-only system
# dependencies (like Vim or graphviz).
# We extract them into the Aptfile.dev file.
COPY Aptfile.dev /tmp/Aptfile.dev
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    $(grep -Ev '^\s*#' /tmp/Aptfile.dev | xargs)

# The production-builder image is responsible for installing dependencies and compiling assets
FROM base as production-builder

# First, we create and configure a dedicated user to run our application
RUN groupadd --gid 1005 my_user \
  && useradd --uid 1005 --gid my_user --shell /bin/bash --create-home my_user
USER my_user
RUN mkdir /home/my_user/app
WORKDIR /home/my_user/app

# Then, we re-configure Bundler
ENV RAILS_ENV=production \
  LANG=C.UTF-8 \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3 \
  BUNDLE_APP_CONFIG=/home/my_user/bundle \
  BUNDLE_PATH=/home/my_user/bundle \
  GEM_HOME=/home/my_user/bundle

# Install Ruby gems
COPY --chown=my_user:my_user Gemfile Gemfile.lock ./
RUN mkdir $BUNDLE_PATH \
  && bundle config --local deployment 'true' \
  && bundle config --local path "${BUNDLE_PATH}" \
  && bundle config --local without 'development test' \
  && bundle config --local clean 'true' \
  && bundle config --local no-cache 'true' \
  && bundle install --jobs=${BUNDLE_JOBS} \
  && rm -rf $BUNDLE_PATH/ruby/${RUBY_VERSION}/cache/* \
  && rm -rf /home/my_user/.bundle/cache/*

# Install JS packages
COPY --chown=my_user:my_user package.json yarn.lock ./
RUN yarn install --check-files

# Copy code
COPY --chown=my_user:my_user . .

# Precompile assets
# NOTE: The command may require adding some environment variables (e.g., SECRET_KEY_BASE) if you're not using
# credentials.
RUN bundle exec rails assets:precompile

# Finally, our production image definition
# NOTE: It's not extending the base image, it's a new one
FROM ruby:$RUBY_VERSION-slim-$DISTRO_NAME AS production

# Production-only dependencies
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  apt-get update -qq \
  && apt-get dist-upgrade -y \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    curl \
    gnupg2 \
    less \
    tzdata \
    time \
    locales \
  && update-locale LANG=C.UTF-8 LC_ALL=C.UTF-8

# Upgrade RubyGems and install the latest Bundler version
RUN gem update --system && \
    gem install bundler

# Create and configure a dedicated user (use the same name as for the production-builder image)
RUN groupadd --gid 1005 my_user \
  && useradd --uid 1005 --gid my_user --shell /bin/bash --create-home my_user
RUN mkdir /home/my_user/app
WORKDIR /home/my_user/app
USER my_user

# Ruby/Rails env configuration
ENV RAILS_ENV=production \
  BUNDLE_APP_CONFIG=/home/my_user/bundle \
  BUNDLE_PATH=/home/my_user/bundle \
  GEM_HOME=/home/my_user/bundle \
  PATH="/home/my_user/app/bin:${PATH}" \
  LANG=C.UTF-8 \
  LC_ALL=C.UTF-8

EXPOSE 3000

# Copy code
COPY --chown=my_user:my_user . .

# Copy artifacts
# 1) Installed gems
COPY --from=production-builder $BUNDLE_PATH $BUNDLE_PATH
# 2) Compiled assets (by Webpacker in this case)
COPY --from=production-builder /home/my_user/app/public/packs /home/my_user/app/public/packs
# 3) We can even copy the Bootsnap cache to speed up our Rails server load!
COPY --chown=my_user:my_user --from=production-builder /home/my_user/app/tmp/cache/bootsnap* /home/my_user/app/tmp/cache/

CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]

Introducing the Ruby on Whales interactive generator

As a bonus, our Ruby on Whales repository ships with a Rails template (published on Rails Bytes), which can help you quickly adopt Docker for development by running a single command (and answering a few questions).

Without further ado, check out the demonstartion below:

An interactive Ruby on Whales installer

You can give it a try by running a single command:

rails app:template LOCATION='https://railsbytes.com/script/z5OsoB'

Acknoledgements

I would like to thank:

Sergey Ponomarev for sharing performance tips and helping battle-test the initial dockerization attempts.
Mikhail Merkushin for his work on Dip.
Dmitriy Nemykin for helping with the major (v2) upgrade.
Oliver Klee (Brain Gourmets) for continuous PRs with the configuration improvements and actualization.

Changelog

2.0.3 (2023-09-21)

Upgrade Node.js installation script.

2.0.2 (2022-11-30)

Use RUN --mount for caching packages between builds instead of manual cleanup.

2.0.1 (2022-03-22)

Replace deprecated apt-key with gpg.

2.0.0 (2022-03-02)

Major upgrade and new chapters.

1.1.4 (2021-10-12)

Added tmp/pids to tmpfs (to deal with “A server is already running” errors).

1.1.3 (2021-03-30)

Updated Dockerfile to mitigate MiniMagic licensing issues. See terraforming-rails#35
Use dictionary to organize environment variables. See terraforming-rails#6

1.1.2 (2021-02-26)

Update dependencies versions. See terraforming-rails#28
Allow to use comments in Aptfile. See terraforming-rails#31
Fix path to Aptfile inside Dockerfile. See terraforming-rails#33

1.1.1 (2020-09-15)

Use .dockerdev directory as build context instead of project directory. See terraforming-rails#26 for details.

1.1.0 (2019-12-10)

Change base Ruby image to slim.
Specify Debian release for Ruby version explicitly and upgrade to buster.
Use standard Bundler path (/usr/local/bundle) instead of /bundle.
Use Docker Compose file format v2.4.
Add health checking to postgres and redis services.