Zero downtime rebranding

So, you have a web application running successfully in production. Suddenly, management decides to rebrand it. For some reasons, cows.com doesn’t cut it anymore, so your client is now a proud owner of www.bulls.com. From a user’s perspective, it should look like a simple redirect in a browser. Your turn to implement it!

“That’s easy,” you may think, “it’s all about reconfiguring an NGINX proxy.” Too bad there isn’t one—a startup you work for had chosen Heroku for hosting a long time ago.

Our task list looks like this:

• Migrate from cows.com to www.bulls.com;
• Enforce the usage of www.bulls.com for web pages (for better SEO);
• Use specific subdomains for API/integrations callbacks;
• Support a bunch of legacy endpoints.

Alas, Heroku has its own way of dealing with configuration: we are restricted to a limited set of options, yet we still need to connect the same application instance to different domains while making sure each of them supports SSL.

Let’s go step by step.

Multiple domain names

We are not able to use CNAME records for our naked domain (cows.com without www) directly because of the way DNS works. So, we need to find a DNS hosting provider that supports ANAME or ALIAS. If the provider that was chosen earlier already supports that—we are in luck, otherwise we need to find a new service to host our brand new domain name.

After we are done with an ALIAS configuration, all we have to do on Heroku is a simple heroku domain:add.

Multiple SSL certificates

Now it’s time to think about SSL certificates. As we want the same application to work on multiple domains and we need to protect them all, we need SNI support. That is a TLS extension that allows a server to present multiple certificates on the same IP address and TCP port—exactly what we need so each of our domains could lead to the same Heroku instance.

Heroku supports SNI with its Heroku SSL offering, but if you happen to use a legacy SSL Endpoint add-on, you will need to upgrade. Luckily, it is not a big deal.

Seems like all we need is to upgrade to SNI and add one SSL certificate per each domain. Unfortunately, with Heroku it is not that straightforward—there is no direct way to add multiple certificates to the same app. There are, however, three proposed workarounds:

1. Generate an SSL certificate with Heroku and Let’s Encrypt with a touch of a button. Works for private applications, but not a good idea if you are dealing with enterprise clients and their sensitive data.
2. Juggle multiple certificates with the SSL Endpoint add-on. Seems like an outdated and messy solution.
3. Use a SAN (Subject Alternative Name) certificate that offers a way to secure multiple domain names at once. You can’t get it for free, but business is business, so we’ll go with this approach.

Requesting a SAN certificate

In order to issue a new certification, we need to generate a private key and prepare a Certificate Signing Request (CSR) with some legal information. You will not be able to use openssl utility’s interactive mode for that—it does not support alternative names. So, we have to compose a correct configuration file by hand. Keep in mind that a certification authority would not let us experiment—if you submit something that does not work, you still have to pay. Luckily, you can try multiple times with free Let’s Encrypt. When you are sure there are no errors in your config, you can send it to your paid service. That is how a final config should look like:

# san.cnf

[req]
default_bits = 2048                 # Current standard
default_md = sha256                 # Current standard
prompt = no                         # Disable interactive mode
req_extensions = req_ext            # Reference additional extensions section
distinguished_name = dn             # Reference distinguished name section

[ dn ]
C = US                              # Country Name
ST = California                     # State
L = San Francisco                   # Locality Name (eg, city)
O = Cows, Inc.                      # Organization Name
OU = Engineering                    # Organizational Unit Name
emailAddress = ceo.cows@example.com # Email address
CN = www.bulls.com                  # Common Name, put your "main" domain name here

[ req_ext ]
subjectAltName = @alt_names         # Reference additional domain names section

[ alt_names ]                       # This is where the magic happens
DNS.1 = bulls.com
DNS.2 = www.cows.com
DNS.3 = cows.com
DNS.4 = api.bulls.com
DNS.5 = callbacks.bulls.com

Now we are ready to generate our keys and a CSR with this one‑liner:

$ openssl req -nodes -new -config san.cnf -keyout bulls.com.key -out bulls.com.csr

It never hurts to double check the result:

$ openssl req -text -noout -verify -in bulls.com.csr

verify OK
Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: C=US, ST=California, L=San Francisco, O=Cows, Inc., OU=Engineering/emailAddress=ceo.cows@example.com, CN=www.bulls.com

        ...

        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name:
                DNS:bulls.com, DNS:www.cows.com, DNS:cows.com, DNS:api.bulls.com, DNS:callbacks.bulls.com

        ....

Two things are important in this output: a “Subject” and a “Subject Alternative Name”. Make sure you have included all your domain names there. Now you can submit the request to your preferred SSL seller and wait for a signed public key. Then send it to Heroku:

$ heroku certs:add bulls.com.crt bulls.com.key

There are few ways to check that everything went well: look for a green lock in your browser, use curl -vI https://www.buls.com or opt for a more bulletproof solution: Qualys SSL Labs checker. It verifies your installation and certificate level and grades it from A to F. Don’t forget to perform a separate check for each domain name.

Handling it all in Rails

Note: If you are using Heroku, most probably your application runs on Rails, if that’s not the case—the information above will still be helpful, but the rest is Ruby-specific.

After we are done with DNS and SSL configurations, we need to reroute all requests coming to the old domain name to the new one, this time—inside our application.

Definitely, we will need some middleware for that. Thanks to rack-rewrite gem, we don’t have to write our own.

Enforcing a “www” subdomain

After installing the gem, we can insert our new middleware into production.rb and tell it to perform a redirect:

# config/environments/production.rb
config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  # Redirect everything from cows.com to www.bulls.com with 301 Moved Permanently
  r301 /.*/, 'http://www.bulls.com$&', host: 'cows.com'
end

That part is a no-brainer, but how about testing it? Enter Rails’ integration testing. Since we are using RSpec, we are going to talk about request specs. Per description, they ”provide a thin wrapper around Rails’ integration tests.”

Request specs allow you to test an application as a whole by executing (almost) real HTTP requests. Under the hood they use Rack::Test::Session.

Every time you call get, post, patch or another verb-related method in your tests, a new Rack request environment object is created and passed to the test session. That request’s env contains some predefined information, including a server’s hostname, which is "www.example.com" by default.

So, every test request is made against "www.example.com". If we want to change that and see how our application handles requests from other hostnames, we can use a method called host!

context "www" do
  it "doesn't redirect if www" do
    # www.example.com is implicit
    get "/users/sign_in"
    expect(response).to be_success
  end

  it "redirects if no www" do
    host! "example.com"
    get "/users/sign_in?utm_tag=test"
    expect(response).to redirect_to("http://www.example.com:3000/users/sign_in?utm_tag=test")
    expect(response.status).to eq 301
  end

  it "redirects legacy domain" do
    host! "legacy.com"
    get "/users/sign_in?utm_tag=test"
    expect(response).to redirect_to("http://www.example.com:3000/users/sign_in?utm_tag=test")
    expect(response.status).to eq 301
  end
end

Note that we use "www.example.com" / "legacy.com" instead of "www.bulls.com" / "cows.com". First of all, there is no good way to override the default hostname as it is just a constant in Rails source code. Secondly, we don’t want to stick to exact names in our specs: we want to test a general logic.

That is how we configure our test environment:

# config/environments/test.rb
config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  r301 /.*/, 'http://www.example.com$&', host: 'legacy.com'
  r301 /.*/, 'http://www.example.com$&', host: 'example.com'
end

Refactoring public endpoints

As it often happens, our imaginary application has a lot of public endpoints to integrate with third-party services: Twilio, Sendgrid, and the like. As it often happens too, there was no logic in the way those endpoints were named: /mail_service_notify, /api/web_hooks/verify, /callbacks/payment, etc.

Why not take the chance to pay back a technical debt? We can introduce a callbacks subdomain that will make it easier to move these functionalities into a separate application later. That way, we will have something like callbacks.bulls.com/third-party/mail/notify.

Of course, we still have to support legacy endpoints. How do we do this without shooting ourselves in the foot? First, we want to make sure that our functionality is covered with request specs and they are all green. Then we update our specs to follow a new naming convention and add some tests for legacy routes:

# Before the second step
describe "mailing service callbacks" do
  let(:request) { post "/mail_service_notify", params }

  it "is successful" do
    is_expected.to be_success
  end
end

# After the second step
describe "mailing service callbacks" do
  let(:request) { post "/third-party/mail/notify", params }

  before { host! "callbacks.example.com" }

  it "is successful" do
    is_expected.to be_success
  end

  context "legacy" do
    let(:request) { post "/mail_service_notify", params }

    it "is successful" do
      host! "legacy.com"
      is_expected.to be_success
    end
  end
end

Tests are now broken, and we are ready to refactor! First, we need new routes and controllers. Note that we are not touching our existing controllers as we still want to test our legacy endpoints. Our tests are green once again, but now we have a lot of code duplication. Let’s drop tests for legacy routes and controllers and replace them with some rewrite rules:

# config/environments/test.rb

# we use the map of legacy->new routes to build a single regexp
LEGACY_CALLBACKS = {
  '/mail_service_notify' => '/third-party/mail/notify',
  '/api/web_hooks/verify' => '/third-party/web_hooks/verify',
  '/callbacks/payment' => '/third-party/payment/status'
}.freeze

config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  rewrite /^(#{LEGACY_CALLBACKS.keys.join('|')}).*/, (lambda do |match, env|
    # Specify the host explicitly
    env['HTTP_HOST'] = 'callbacks.bulls.com'
    # Replace matching prefix with the corresponding new namespace
    replace_path = LEGACY_CALLBACKS[match[1]]
    match[0].sub(match[1], replace_path).to_s
  end), host: 'cows.com'
end

Now if our tests are green—rewrite rules work as expected!

Dealing with environments

You have probably noticed that we had to duplicate our redirect and rewrite rules for each environment, and even within the same environment (for different hosts). Can’t we just re-use our rewrite rules? Sadly, there are no drop-in solutions, but we have developed one of our own: a simple сlass called RackRewriteConfig. It allows you to write dynamic rules, like so:

# config/legacy_routes.rb
class LegacyRoutes < RackRewriteConfig
  LEGACY_CALLBACKS = {
    '/mail_service_notify' => '/third-party/mail/notify',
    '/api/web_hooks/verify' => '/third-party/web_hooks/verify',
    '/callbacks/payment' => '/third-party/payment/status'
  }.freeze

  # Domain redirect rule
  configure(:domain_redirect) do |host:, target:|
    r301 /.*/, "#{target}$&", host: host, method: :get
  end

  # Rewrite rules for callbacks
  configure(:legacy_callbacks) do |host:, target:|
    rewrite /^(#{LEGACY_CALLBACKS.keys.join('|')}).*/, (lambda do |match, env|
      env['HTTP_HOST'] = target
      # Replace matching prefix with the corresponding new namespace
      replace_path = LEGACY_CALLBACKS[match[1]]
      match[0].sub(match[1], replace_path).to_s
    end), host: host
  end
end

Then we can apply the same set of rules to each environment with different parameters:

# config/environments/production.rb
config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  LegacyRoutes.apply(self, :domain_redirect, host: 'bulls.com', target: 'https://www.bulls.com')
  LegacyRoutes.apply(self, :legacy_callbacks, host: 'cows.com', target: 'callbacks.bulls.com')
  LegacyRoutes.apply(self, :legacy_callbacks, host: 'www.cows.com', target: 'callbacks.bulls.com')
  LegacyRoutes.apply(self, :domain_redirect, host: 'cows.com', target: 'https://www.bulls.com')
  LegacyRoutes.apply(self, :domain_redirect, host: 'www.cows.com', target: 'https://www.bulls.com')
end

# config/environments/test.rb
config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  LegacyRoutes.apply(self, :domain_redirect, host: 'example.com', target: 'http://www.example.com:3000')
  LegacyRoutes.apply(self, :legacy_callbacks, host: 'legacy.com', target: 'callbacks.example.com')
  LegacyRoutes.apply(self, :legacy_callbacks, host: 'www.legacy.com', target: 'callbacks.example.com')
  LegacyRoutes.apply(self, :domain_redirect, host: 'legacy.com', target: 'http://www.example.com:3000')
  LegacyRoutes.apply(self, :domain_redirect, host: 'www.legacy.com', target: 'http://www.example.com:3000')
end

Now we can finally deploy our changes to production and announce a new name for our product!

We have solved all our issues with multiple domains and multiple SSL certificates and our code is now ready to handle the new behavior. Our goal has been achieved: no downtime, no missed requests, and no dissatisfied customers, even on a zero-ops platform as a service such as Heroku.