The tale of Sprockets and Webpacker duality

One requirement for keeping an application in a healthy state is upgrading dependencies. Some upgrades are no-brainers to implement, while others require some more preliminary work, especially when performing a major version upgrade (you’ve probably heard about Rails dual boot strategy). Additionally, there is also a special kind of “upgrade”—migrating from one engine to another. For example, switching from Sprockets to Webpack(er).

This was exactly our situation with a project we worked on. We had to deal with a monolithic Rails app with all the Rails Way goodies: HTML-first, with JavaScript/CoffeeScript/jQuery sprinkles and SCSS (Bootstrap) styling. We needed a way to migrate seamlessly from Sprockets to Webpacker. Due to its frontend “architecture”, it would’ve been nearly impossible to migrate gradually; everything would need to be updated at once.

Most applications perform asset manager migration in a single step: simply merge the “Switch to Webpacker” PR and deploy. Of course, even though a lot of QA and testing takes place in the staging environments, it’s still hard to be 100% confident that everything will be fine after making the switch. When an application has millions of daily users, bugs cost a lot, and performing atomic upgrades could be too risky.

We had to find a safer way. After some thinking, we came up with the idea of a dual build that we’d like to present today.

Dual build: an overview

The idea behind the dual build is pretty similar to a dual boot, where the application code is made compatible with two different versions of a framework. For our dual build, we have two different assets builders, and we want them to use the same source code and produce equivalent bundles.

In a nutshell, here’s how it works:

• As before, assets are kept in the app/assets folder.
• Webpack entry points (from app/javascripts) also import source code from app/assets.
• NPM packages are added to mirror vendor/assets and assets from gems.
• Running rails assets:precompile generates both public/assets and public/packs.
• A feature toggle is added to control when to use Webpacker or Sprockets (For example, Rails helpers, such as javascript_pack_tag, are overridden to fallback to javascript_include_tag if the user disables the Webpacker feature).

We could use this setup to switch any individual user from Sprockets to Webpacker, thus, we’re able to gradually rollout this feature or momentarily perform a rollback if something goes wrong.

During rollout, we caught a few bugs which could have resulted in a bad user experience → unsatisfied customers → the technical support team overwhelmed by tickets. Although this dual build approach might seem overly complicated, it quite literally saved the money and the nerves of our client and their customers. So, it really did pay off.

Now, let’s move on to the technical details and actually talk about how to set up a dual build for a Rails application.

Preliminary work

There were a couple of things that we did before adding Webpacker to the equation.

First, we checked that all third-party dependencies had corresponding NPM versions. If something was missing, we upgraded/migrated to a different library/version that was also available on NPM. A good ol’ Excel spreadsheet helped us keep track of this task 🙂.

Following this, we got rid of CoffeeScript via decaffeinate. To that end, we converted all the .coffee files into ES6:

npx decaffeinate app/assets/javascripts

Then, because of Sprockets’ support requirements, we had to convert our shiny ES6 code to ES5:

npx babel app/assets/javascripts --out-dir app/assets/javascripts

To run the command above, we needed to have babel modules installed in our app. When using Webpacker, it’s already installed.

Finally, we applied the Prettier formatting like so:

npx prettier --write app/assets/javascripts

After these steps were complete, we reviewed the resulting modules, removed some remaining technical comments left by decaffeinate, and ensured everything was working correctly.

Webpackerizing JavaScript

We started our experiment by making the JS scripts Webpack-compatible. To do this, we ran rails webpacker:install and added our very first entry point (admin.js):

// app/javascript/packs/admin.js

import '../../assets/javascripts/admin';

All we needed to do was import our existing Sprockets bundle. Let’s take a look it:

// app/assets/javascripts/admin.js

//= require services/s3_service
//= require components/uploaders/input_uploader
//= require admin/index.js

It only contained Sprockets require statements. If we ran rails webpacker:compile, we’d get a no-op packs/admin-<sha>.js script, since there is no JS content and Webpack doesn’t understand // require. We needed to use imports like this:

// app/assets/javascripts/admin.js

import './services/s3_service';
import './components/uploaders/input_uploader';
import './admin/index';

Good! This made webpacker:compile work. Unfortunately, it also broke our existing Sprockets bundle 😞.

After some more brainstorming we came up with the following idea:

• Leave the // require statements as they are since they’re ignored by Webpack (because they’re just comments).
• Teach Sprockets to ignore import statements.

Luckily, Sprockets has a pluggable architecture which allows us to inject custom code to preprocess source files before bundling them. For example, we can delete lines that have // webpack-only comments:

// app/assets/javascripts/admin.js

//= require services/s3_service
//= require components/uploaders/input_uploader
//= require admin/index.js

import './services/s3_service'; // webpack-only
import './components/uploaders/input_uploader'; // webpack-only
import './admin/index'; // webpack-only

With that, the code above is the final composition of our dual entry point!

And here is our WebpackOnlyPreprocessor:

# config/initializers/assets.rb

# ...

# Strip Webpack-specific lines from JS
module Sprockets::WebpackOnlyProcessor
  class << self
    def call(input)
      data = input[:data].gsub(%r{^.*// webpack-only\s*$}, "")
      data.gsub!(%r{^\s*// webpack-only-begin.*^\s*// webpack-only-end\s*$}m, "")
      { data: data }
    end
  end
end

Rails.application.config.assets.configure do |env|
  env.register_preprocessor "application/javascript", Sprockets::WebpackOnlyProcessor
end

The plugin above also supports excluding blocks of code:

// webpack-only-begin
import './services/s3_service';
import './components/uploaders/input_uploader';
import './admin/index';
// webpack-only-end

Finally, we replaced javascript_include_tag with a slightly patched javascript_pack_tag to serve both versions depending on the current user:

def javascript_pack_tag(*args)
  if Rails.configuration.x.webpack_enabled ||
     (defined?(user_features) && user_features.enabled?("webpacker"))
    return super
  end

  javascript_include_tag(*args)
end

Dealing with Sass

Since Sass uses @import instead of // require, it worked correctly out of the box.
However, we did have a few places where assets were imported via Sprockets. For example, Bootstrap:

// = require 'bootstrap/functions'
// = require 'bootstrap/variables'
// = require 'bootstrap/mixins'

Simply replacing these declarations with imports works in the Sprockets environment but it breaks in Webpacker because different paths are used. We resolved this by adding aliases to our Webpack config:

// config/webpack/aliases.js
const { resolve } = require('path');

const config = {
  resolve: {
    alias: {
      'bootstrap/functions': resolve(__dirname, '..', '..', 'node_modules/bootstrap/scss/_functions.scss'),
      'bootstrap/variables': resolve(__dirname, '..', '..', 'node_modules/bootstrap/scss/_variables.scss'),
      'bootstrap/mixins': resolve(__dirname, '..', '..', 'node_modules/bootstrap/scss/_mixins.scss')
    }
  }
};

module.exports = config;

// config/webpack/environment.js
// ...
const aliasesConfig = require('./aliases');
environment.config.merge(aliasesConfig);
// ...

The real challenge with our Sass dual build was dealing with asset urls. We couldn’t use image-url or font-url anymore, and had to make due with just plain url. After some trial and error, we found a way to hack the Sass compiler used by Sprockets to automagically resolve asset paths:

# config/initializers/assets.rb

require "sprockets/sass_processor"

module Sprockets
  module SassProcessor::Functions
    # Override url to handle asset pipeline backed files
    def url(path)
      path = path.value
      url =
        if options[:sprockets] && sprockets_context.environment.resolve(path)
          sprockets_context.compute_asset_path(path, skip_pipeline: true)
        else
          path
        end
      ::SassC::Script::Value::String.new("url(\"#{url}\")")
    end
  end

  class SassCompressor
    # Fix #call to pass sprockets context as option
    # Original: https://github.com/sass/sassc-rails/blob/8d0462d54b5b5dd84cb1df31823c3afaebb64534/lib/sassc/rails/compressor.rb#L17-L30
    def call(input)
      SassC::Engine.new(
        input[:data],
        @options.merge({
                         style: :compressed,
                         sprockets: {
                           # Source https://github.com/rails/sprockets/blob/2f7b7e5e67f47c32a2d637b7e90dfa5ecf922eb3/lib/sprockets/sass_processor.rb#L56-L69
                           context: input[:environment].context_class.new(input)
                         }
                       })
      ).render
    end
  end
end

For Webpack, we had to use aliases again for all the assets:

// config/webpack/aliases.js

const config = {
  resolve: {
    alias: {
      // ...
      './logo-small-dark-transparent.png': abs('app/assets/images/logo-small-dark-transparent.png'),
      './logo-small-dark-transparent@2X.png': abs('app/assets/images/logo-small-dark-transparent@2X.png'),
      './fa/font-awesome.woff2': abs('app/assets/fonts/fa/font-awesome.woff2')
    }
  }
}

Yeah, so, this was a bit of a dirty hack, but we were okay with it, since we only needed it for the migration period.

The verdict on dual build: yes or no?

The initial setup itself was pretty quick (it took two engineers a week or so of work). As always, achieving 90% of the goal was much easier than fixing the remaining 10%: exposing globals, improving CI and production (Heroku) build times, and so on.

Is it all worth it? Definitely, yes. Well, if you don’t want to turn your users into beta-testers, anyway 🙂

Hopefully this article helped you come to grips a bit more with the nature of duality—at least in terms of creating a dual build with Sprockets and Webpacker. With luck, this method will prove useful if you need a safer upgrade option, should you choose.