Anyway Config: Keep your Ruby configuration sane

Configuration is one of the most critical markers of codebase health: the more your application grows and matures, the harder it is to deal with API keys, .env files, and other settings. In my RailsConf 2019 talk “Terraforming legacy Rails applications” (slides, video), I talked about “ENV Hell”—something I’m sure most of the readers working on larger Ruby apps are familiar with.

Do you live in ENV Hell?

The most popular configuration pattern in Rails apps is to store all values in the .env file and to load them into process environment on application start (with dotenv or dotenv-rails gems), so they can later be accessible with ENV["KEY"] from code.

That pattern itself comes from a good place: one of the well-respected twelve-factor principles states “Store config in the environment.”

However, let’s perform a simple experiment. Go to your shell, navigate to the project that has a .env file in the root folder, and execute this command:

cat .env | grep '[^\s]' | wc -l
  52 # That's the average value we see in mature projects we are invited to work on

What’s the number? Is it in the order of dozens? Then my condolences, you live in the ENV Hell. It’s all right. We’ve all been there.

Here’s how ENV Hell usually feels like:

• .env file grows and becomes barely understandable;
• .env.sample goes out of sync causing hard-to-debug failures during local development;
• ENV is a global state representing the outer world, which makes debugging and especially testing harder.

These problems usually creep into deployment setup: for example, Heroku apps tend to have hundreds of environment variables in their configuration. You can quickly check this by running:

$ heroku config -a legacy-project | wc -l

  131

Now, take this quick survey to assess the situation in your current Rails project:

1. Is it possible to launch rails s right after the project has been bootstrapped (bin/setup or similar)?
2. In addition to rails s, do tests pass?
3. Do you know where to get credentials if the application is missing them?
4. Is there a clear workflow for adding new values to configuration?
5. Is it possible to use personal secrets (like third-party API tokens) without changing the application code?

The more no’s you have, the quicker you should take action and reconsider your approach to configuration.

Stay tuned to find out how.

Secrets and Settings: two types of configuration parameters

Putting all eggs (configuration parameters) into one basket (.env file) has one major downside: we lose the information about the nature of our values. We mix them all: sensitive and non-sensitive; business logic- and framework- related.

To become better at keeping track of all the configuration values—we could mentally split them into Settings and Secrets. Let’s see how they differ.

Settings are internal

Settings modify technical characteristics and framework settings, for example: WEB_CONCURRENCY, RAILS_MAX_THREADS, RAILS_SERVE_STATIC_FILES. Let’s call them framework settings, where “framework” is not only Rails but all parts of our stack, such as Puma and Sidekiq, for example.

There are also application settings that change the application behavior as a whole, such as global feature toggles (CHAT_ENABLED=1) or flags to enable dev tools (GRAPHIQL_ENABLED=1).

Ideal Settings have the following properties:

• They must have sensible defaults for development and tests.
• They could be overridden in production via environment variables (Heroku sets RAILS_SERVE_STATIC_FILES for you).
• They could be stored as plain text in the repo.
• They could even be hardcoded in environment-specific configuration files (config/environments/development.rb).

Secrets are external

Secrets carry the information required to interact with other systems and services. They also could be grouped into two types: system and service.

System secrets include access credentials for the essential parts of the infrastructure, such as databases (DATABASE_URL) and cache servers (REDIS_URL). Heroku add-ons, for example, set these values for you in production, you only need to take care of them in development (we put them into docker-compose.yml).

The second group, service secrets, contains the credentials of the third-party services (API keys, tokens, whatever).

Not all the information that counts as a secret has to be sensitive: think API hostnames and limit values.

There is one important technical difference between system and service secrets:

The application must fail on boot if a system secret is missing or invalid

That should not be true for service secrets—not every piece of configuration is essential for your application to start.

Active Storage example

Say, you worked on a file uploading feature and decided to use Active Storage with the Amazon S3 backend. Here are the storage.yml and configuration files that have been merged to master (code comes from a real-life commit):

# config/application.rb
config.active_storage.service = :s3

# config/environments/test.rb
config.active_storage.service = :local

# config/storage.yml
local:
  service: Disk
  root: <%= Rails.root.join("tmp/storage") %>

s3:
  service: S3
  access_key_id: <%= ENV["AWS_ACCESS_KEY_ID"] %>
  secret_access_key: <%= ENV["AWS_SECRET_ACCESS_KEY"] %>
  region: us-east-1
  bucket: ENV["AWS_BUCKET"]

Now everyone on your team who has not set Amazon-related ENV variables won’t be able to run the application even in development. We are artificially creating a barrier for entry, even though directly using AWS SDK locally is not required: Active Storage can use :local setting not only for tests, but for development too. One may say: “I want to have my local environment as close to production as possible!” That’s a good intention, but should it be a hard requirement?

Imagine we could make using real AWS buckets for development an opt-in feature.

# config/application.rb
config.active_storage.service =
  if AWSConfig.storage_configured?
    $stdout.puts "Using :s3 service for Active Storage"
    :s3
  else
    :local
  end

All right, that seems foreign and does not come out of the box with Rails. What is AWSConfig, and how does it know that it’s configured? That is the Anyway Config gem in action, the one I’ve been taunting you with since the beginning of this article.

Enter Anyway Config

Besides using ENV for storing configuration data, modern Rails gives you plenty of other options: from directly editing parameters in config/initializers to using plain old YML files and, since Rails 5.2, encrypted credentials that are safe to check into source control.

Here are the golden configuration rules we try to follow at Evil Martians:

• Store sensitive information in Rails credentials (each environment has its own *.enc file).
• Keep non-sensitive information in named YAML configs.
• Allow overriding any value via ENV.
• Store local (development) secrets and settings in *.local.yml and credentials/local.yml.enc files.
• If you need to share extra-sensitive credentials with all your team members, use centralized encrypted storage. Keybase does this job for us.

It is hard to avoid the embarrassment of riches: using all these different ways to get configuration into your app adds to the cognitive overhead. So we came up with a tool that provides a standard, pure Ruby interface to all configuration settings.

Anyway Config is a gem that allows you to manage different sources of data transparently. Moreover, it makes your code independent of the way you store your settings by introducing configuration classes. No more Rails.credentials, Rails.application.config_for, or ENV calls; you only need to deal with Ruby classes.

The gem has a long story: extracted initially from the first gem of mine, Influxer, it has been used mostly in libraries (for instance, AnyCable) for a long time. The recent 2.0 release is heavily inspired by the application development use-cases we had in Evil Martians in the last couple of years.

Let’s return to our Active Storage example to see how we could have configured it with Anyway Config.

Adding anyway_config to your Gemfile in Rails gives you access to handy generators that create new configuration classes:

$ rails generate anyway:config aws access_key_id secret_access_key region storage_bucket
    generate  anyway:install
       rails  generate anyway:install
      create  config/configs/application_config.rb
      append  .gitignore
      insert  config/application.rb
      create  config/configs/aws_config.rb
Would you like to generate a aws.yml file? (Y/n) n

That would add two files to your project:

• config/configs/application_config.rb—base class for all configuration classes (only created if not exists):

# Base abstract class for config files.
# It provides the `instance` method, which returns the default
# instance for this config.
#
# It also delegates all the missing methods to this instance,
# thus allowing you to use the class itself as a singleton config instance.
class ApplicationConfig < Anyway::Config
  class << self
    delegate_missing_to :instance

    def instance
      @instance ||= new
    end
  end
end

• config/configs/aws_config.rb—AWS configuration class:

class AWSConfig < ApplicationConfig
  # attr_config defines readers and writers
  # for configuration parameters
  attr_config :access_key_id, :secret_access_key,
              :region, :storage_bucket
end

Note that you need to configure Rails inflector to understand “AWS” by adding an acronym to config/initializers/inflections.rb:

ActiveSupport::Inflector.inflections do |inflect|
  # ...
  inflect.acronym "AWS"
end

If you answer “yes” to the generator prompt, config/aws.yml file will be added as well. For now, we don’t want to store any information in plain text; we’re going to use credentials.

Let’s edit our configuration class and add the default value for the region as well as the #storage_configured? method:

class AWSConfig < ApplicationConfig
  # We can provide default values by passing a Hash
  attr_config :access_key_id, :secret_access_key,
              :storage_bucket, region: "us-east-1"

  def storage_configured?
    access_key_id.present? &&
      secret_access_key.present? &&
      storage_bucket.present?
  end
end

Then, we need to populate values for production. Let’s open our credentials file and define the following values:

$ RAILS_MASTER_KEY=<production key> rails credentials:edit -e production

aws:
  access_key_id: "secret"
  secret_access_key: "very-very-secret"
  storage_bucket: "also-could-be-a-secret"

Depending on your use-case, you may not consider storage_bucket to be a piece of sensitive information. If that’s the case, you can define it in config/aws.yml:

# config/aws.yml
production:
  aws:
    storage_bucket: my-public-bucket

At any time, you can override any value by providing the corresponding environment variable:

AWS_STORAGE_BUCKET=another-bucket rails s

Did you notice that your code doesn’t care about where the configuration values come from and only knows about the AWSConfig class? That’s the main benefit of this approach.

If one day you decide to use AWS locally, you can do that by putting your personal credentials to config/aws.local.yml. If you’re worried about storing secrets as plain text on your machine, you can use local Rails credentials:

rails credentials:edit -e local

Anyway Config will assign higher priority to your local data.

And as a bonus (especially useful in production), you can track the source of every value:

AWSConfig.to_source_trace
# =>
# {
#  "access_key_id" => {value: "XYZ", source: {type: :credentials, store: "config/credentials/production.yml.enc"}},
#  "secret_access_key" => {value: "123KLM", source: {type: :credentials, store: "config/credentials/production.yml.enc"}},
#  "region" => {value: "us-east-1", source: {type: :defaults}},
#  "storage_bucket" => {value: "example-bucket", source: {type: :yml, path: "config/aws.yml"}}
#  }

You can also pretty-print it to get a more human-friendly output:

pp AWSConfig.new
# =>
# #<AWSConfig
#  config_name="aws"
#  env_prefix="AWS"
#  values:
#    access_key_id => "XYZ" (type=credentials store=config/credentials/production.yml.enc)
#    secret_access_key => "123KLM" (type=credentials store=config/credentials/production.yml.enc)
#    region => "us-east-1" (type=defaults)
#    storage_bucket => "my-public-bucket" (type=yml path=config/aws.yml)

To sum up, Anyway Config gives you:

• Configuration classes instead of different data source wrappers.
• Support for local secrets and settings.
• Separate configuration files instead of a single bloated .env or application.yml.

As Ruby application grows, configuration management can quickly become a nightmare. You can keep it under control by paying more attention to how you organize configuration files and how you treat different kinds of values.

Making codebase free of knowledge of every particular configuration data source also helps to keep your project healthy. Anyway Config provides you with a common abstraction that suits all cases—now you are free to mix, match, and override pieces of configuration coming from different sources without any cognitive overhead.

Use ENV responsibly!

By the way, if you’re looking into “terraforming” your mature Rails application to introduce best practices to date: from setting up containerized development environment to speeding up tests, adopting GraphQL, optimizing database queries or generally getting rid of any performance bottlenecks—feel free to drop us a line, my colleagues and I can definitely help.