Active Storage meets GraphQL: Direct Uploads

Happy code for happy people

Let me start with a strong statement: I am a happy person.

This happiness is multi-dimensional, with some dimensions bringing more value than others (and I’m not talking about the most valuable ones today, sorry).

For example, I’m happy at work because I finally got an opportunity to build a project from scratch using such “cutting-edge” (if we can say so about 15-years old framework 😉) technologies, as Rails 6.

Believe it or not, I’ve been working on Rails production projects for 5 years, and haven’t even touched Rails 5, only Rails 4!

“Legacy Rails applications” was a kind of my specialization (by the way, that’s the topic of my upcoming RailsConf session).

These dark times have ended this January: I ran gem install rails --prerelease && rails new ***.

(Actually, it was rails new *** -d postgresql --skip-action-mailbox --skip-action-text --skip-action-cable --skip-sprockets --skip-spring --skip-test --skip-bundle.)

The project I’m working on is not a 100% new codebase; it has a lot of code initially written for Rails 4.

And it has GraphQL API as the main entry point for clients (web and mobile applications).

As a part of porting the old codebase into the new app, we migrated from CarrierWave to Active Storage. The experience was smooth. And though Active Storage has some missing parts, it has its advantages and a Rails-way simplicity.

NOTE: If you’re new to Active Storage, check out the post I wrote a year ago with my colleague: “Rails 5.2: Active Storage and beyond”.

So, it’s time to move to the most notable advantage of Active Storage: direct uploads implemented out-of-the-box.

Life before Active Storage

First, let me tell you how we dealt with file uploads in Rails 4. Neither GraphQL specification nor graphql Ruby gem specifies a way to cook file uploads properly.

There is an open source specification, which has implementations in different languages, including Ruby. It “describes” the Upload scalar type, does some Rack middleware magic to pass uploaded files as variables and kinda works transparently.

Sounds like a “plug-n-play.” In theory. In practice, it transformed into “plug-n-play-n-fail-n-fix-n-fail-n-fix”:

• Buggy client implementations (especially for React Native)
• Side-effects due to a non-strict Upload type (which doesn’t care about the actual object type)
• Apollo dependency (yes, we said “Good-bye!” to Apollo in the new version; but that’s another story).

No surprises (and no alarms 😉), we decided to get rid of this hack and use a good old REST for uploading files.

And here comes Active Storage with direct uploads.

Directing uploads 🎥

What is “direct upload” by the way?

This term is usually used in conjunction with cloud storage services (e.g., Amazon S3) and means the following: instead of uploading a file using the API server, the client uploads it directly to the cloud storage using credentials generated by the API server.

Good news—Active Storage provides a server-side API to handle direct uploads and a frontend JS client out-of-the-box.

Another good news—this API is abstract, works with any service supported by Active Storage (i.e., filesystem, S3, GCloud, Azure). And that’s great: you can use filesystem locally and S3 in production without all that if-s and else-s.

Good news rarely comes without bad news, though. And the bad news is that Active Storage (and Rails in general) does not know anything about GraphQL and relies on its own REST API for retrieving direct upload credentials.

What do we need to make it all happen in GraphQL?

First of all, an ability to get direct upload credentials using GraphQL API (via mutation).

Secondly, it would be great to re-use as much JavaScript code from a framework as possible to avoid re-inventing the wheel.

createDirectUpload mutation…

Unfortunately, Rails doesn’t have any documentation for the server-side direct uploads implementation.

All we have is the source code for DirectUploadsController:

def create
  blob = ActiveStorage::Blob.create_before_direct_upload!(blob_args)
  render json: direct_upload_json(blob)
end

private

def blob_args
  params.require(:blob).permit(:filename, :byte_size, :checksum, :content_type, :metadata).to_h.symbolize_keys
end

def direct_upload_json(blob)
  blob.as_json(root: false, methods: :signed_id).merge(
    direct_upload: {
      url: blob.service_url_for_direct_upload,
      headers: blob.service_headers_for_direct_upload
    }
  )
end

Take a look at the checksum parameter: this is one of mane hidden gems of Active Storage—a built-in file contents verification.

When a client requests a direct upload, it can specify the checksum of the file (MD5 hash encoded as base64), and the service (e.g., Active Storage itself or S3) will use this checksum later to verify the uploaded file contents.

Let’s come back to GraphQL.

GraphQL mutations are pretty much like Rails controllers, so transforming the above code into a mutation is a straightforward:

class CreateDirectUpload < GraphQL::Schema::Mutation
  class CreateDirectUploadInput < GraphQL::Schema::InputObject
    description "File information required to prepare a direct upload"

    argument :filename, String, "Original file name", required: true
    argument :byte_size, Int, "File size (bytes)", required: true
    argument :checksum, String, "MD5 file checksum as base64", required: true
    argument :content_type, String, "File content type", required: true
  end

  argument :input, CreateDirectUploadInput, required: true

  class DirectUpload < GraphQL::Schema::Object
    description "Represents direct upload credentials"

    field :url, String, "Upload URL", null: false
    field :headers, String,
          "HTTP request headers (JSON-encoded)",
          null: false
    field :blob_id, ID, "Created blob record ID", null: false
    field :signed_blob_id, ID,
          "Created blob record signed ID",
          null: false
  end

  field :direct_upload, DirectUpload, null: false

  def resolve(input:)
    blob = ActiveStorage::Blob.create_before_direct_upload!(input.to_h)

    {
      direct_upload: {
        url: blob.service_url_for_direct_upload,
        # NOTE: we pass headers as JSON since they have no schema
        headers: blob.service_headers_for_direct_upload.to_json,
        blob_id: blob.id,
        signed_blob_id: blob.signed_id
      }
    }
  end
end

# add this mutation to your Mutation type
field :create_direct_upload, mutation: CreateDirectUpload

Now, to retrieve a direct upload payload from the server, your GraphQL client must perform the following request:

mutation {
 createDirectUpload(input: {
   filename: "dev.to", # file name
   contentType: "image/jpeg", # file content type
   checksum: "Z3Yzc2Q5iA5eXIgeTJn", # checksum
   byteSize: 2019 # size in bytes
 }) {
   directUpload {
     signedBlobId
   }
 }
}

…and some JavaScript

Disclaimer: the JS implementation below is just a sketch and hasn’t been tested in reality (since in my project we don’t use any of Rails’ JS code). All I checked for is that it compiles.

To upload a file, a client must perform the following steps:

• Obtain the file metadata (filename, size, content type and checksum)
• Request direct upload credentials and a blob ID via API—createDirectUpload mutation
• Upload the file using the credentials (no GraphQL involved, HTTP PUT request).

For step 1 and 3 we can re-use some of the code from the JS library that comes along with Rails (don’t forget to add "@rails/activestorage" to your package.json).

Let’s write a getFileMetadata function:

import { FileChecksum } from "@rails/activestorage/src/file_checksum";

function calculateChecksum(file) {
  return new Promise((resolve, reject) => {
    FileChecksum.create(file, (error, checksum) => {
      if (error) {
        reject(error);
        return;
      }

      resolve(checksum);
    });
  });
}


export const getFileMetadata = (file) => {
  return new Promise((resolve) => {
    calculateChecksum(file).then((checksum) => {
      resolve({
        checksum,
        filename: file.name,
        content_type: file.type,
        byte_size: file.size
      });
    });
  });
};

FileChecksum class is responsible for calculating the required checksum and is used by Active Storage in DirectUpload class.

Now you can use this function to build your GraphQL query payload:

// pseudo code
getFileMetadata(file).then((input) => {
  return performQuery(
    CREATE_DIRECT_UPLOAD_QUERY,
    variables: { input }
  );
});

Now it is time to write a function to upload files directly to the storage service!

import { BlobUpload } from "@rails/activestorage/src/blob_upload";

export const directUpload = (url, headers, file) => {
  const upload = new BlobUpload({ file, directUploadData: { url, headers } });
  return new Promise((resolve, reject) => {
    upload.create(error => {
      if (error) {
        reject(error);
      } else {
        resolve();
      }
    })
  });
};

Our complete client side code example would be the following:

getFileMetadata(file).then((input) => {
  return performQuery(
    CREATE_DIRECT_UPLOAD_QUERY,
    variables: { input }
  ).then(({ directUpload: { url, headers, signedBlobId }) => {
    return directUpload(url, JSON.parse(headers), file).then(() => {
      // do smth with signedBlobId—our file has been uploaded!
    });
  });
});

Looks like we did it! Hope that helps you to build your awesome new Rails + GraphQL project)

For a more realistic example, check out this snippet from our React Native application.

In Conclusion, or what to do with signedBlobId

Let me provide just a quick example of how we use signed blob IDs in the app—the attachProfileAvatar mutation:

class AttachProfileAvatar < GraphQL::Schema::Mutation
  description <<~DESC
    Update the current user's avatar
    (by attaching a blob via signed ID)
  DESC

  argument :blob_id, String,
           "Signed blob ID generated via `createDirectUpload` mutation",
           required: true

  field :user, Types::User, null: true

  def resolve(blob_id:)
    # Active Storage retrieves the blob data from DB
    # using a signed_id and associates the blob with the attachment (avatar)
    current_user.avatar.attach(blob_id)
    { user: current_user }
  end
end

That’s it!

Active Storage meets GraphQL pt. 2: exposing attachment URLs

Active Storage meets GraphQL pt. 2: exposing attachment URLs

April 23, 2019

Read also