Active Storage meets GraphQL: Direct Uploads
Other parts:
- Active Storage meets GraphQL: Direct Uploads
- Active Storage meets GraphQL pt. 2: exposing attachment URLs
Happy code for happy people
Let me start with a strong statement: I am a happy person.
This happiness is multi-dimensional, with some dimensions bringing more value than others (and I’m not talking about the most valuable ones today, sorry).
For example, I’m happy at work because I finally got an opportunity to build a project from scratch using such “cutting-edge” (if we can say so about 15-years old framework 😉) technologies, as Rails 6.
Believe it or not, I’ve been working on Rails production projects for 5 years, and haven’t even touched Rails 5, only Rails 4!
“Legacy Rails applications” was a kind of my specialization (by the way, that’s the topic of my upcoming RailsConf session).
These dark times have ended this January: I ran gem install rails --prerelease && rails new ***.
(Actually, it was rails new *** -d postgresql --skip-action-mailbox --skip-action-text --skip-action-cable --skip-sprockets --skip-spring --skip-test --skip-bundle.)
The project I’m working on is not a 100% new codebase; it has a lot of code initially written for Rails 4.
And it has GraphQL API as the main entry point for clients (web and mobile applications).
As a part of porting the old codebase into the new app, we migrated from CarrierWave to Active Storage. The experience was smooth. And though Active Storage has some missing parts, it has its advantages and a Rails-way simplicity.
NOTE: If you’re new to Active Storage, check out the post I wrote a year ago with my colleague: “Rails 5.2: Active Storage and beyond”.
So, it’s time to move to the most notable advantage of Active Storage: direct uploads implemented out-of-the-box.
Life before Active Storage
First, let me tell you how we dealt with file uploads in Rails 4. Neither GraphQL specification nor graphql Ruby gem specifies a way to cook file uploads properly.
There is an open source specification, which has implementations in different languages, including Ruby. It “describes” the Upload scalar type, does some Rack middleware magic to pass uploaded files as variables and kinda works transparently.
Sounds like a “plug-n-play.” In theory. In practice, it transformed into “plug-n-play-n-fail-n-fix-n-fail-n-fix”:
- Buggy client implementations (especially for React Native)
- Side-effects due to a non-strict
Uploadtype (which doesn’t care about the actual object type) - Apollo dependency (yes, we said “Good-bye!” to Apollo in the new version; but that’s another story).
No surprises (and no alarms 😉), we decided to get rid of this hack and use a good old REST for uploading files.
And here comes Active Storage with direct uploads.
Directing uploads 🎥
What is “direct upload” by the way?
This term is usually used in conjunction with cloud storage services (e.g., Amazon S3) and means the following: instead of uploading a file using the API server, the client uploads it directly to the cloud storage using credentials generated by the API server.
Direct uploads diagram
Good news—Active Storage provides a server-side API to handle direct uploads and a frontend JS client out-of-the-box.
Another good news—this API is abstract, works with any service supported by Active Storage (i.e., filesystem, S3, GCloud, Azure). And that’s great: you can use filesystem locally and S3 in production without all that if-s and else-s.
Good news rarely comes without bad news, though. And the bad news is that Active Storage (and Rails in general) does not know anything about GraphQL and relies on its own REST API for retrieving direct upload credentials.
What do we need to make it all happen in GraphQL?
First of all, an ability to get direct upload credentials using GraphQL API (via mutation).
Secondly, it would be great to re-use as much JavaScript code from a framework as possible to avoid re-inventing the wheel.
createDirectUpload mutation…
mutation preview in GraphiQL
Unfortunately, Rails doesn’t have any documentation for the server-side direct uploads implementation.
All we have is the source code for DirectUploadsController:
def create
blob = ActiveStorage::Blob.create_before_direct_upload!(blob_args)
render json: direct_upload_json(blob)
end
private
def blob_args
params.require(:blob).permit(:filename, :byte_size, :checksum, :content_type, :metadata).to_h.symbolize_keys
end
def direct_upload_json(blob)
blob.as_json(root: false, methods: :signed_id).merge(
direct_upload: {
url: blob.service_url_for_direct_upload,
headers: blob.service_headers_for_direct_upload
}
)
endTake a look at the checksum parameter: this is one of mane hidden gems of Active Storage—a built-in file contents verification.
When a client requests a direct upload, it can specify the checksum of the file (MD5 hash encoded as base64), and the service (e.g., Active Storage itself or S3) will use this checksum later to verify the uploaded file contents.
Let’s come back to GraphQL.
GraphQL mutations are pretty much like Rails controllers, so transforming the above code into a mutation is a straightforward:
class CreateDirectUpload < GraphQL::Schema::Mutation
class CreateDirectUploadInput < GraphQL::Schema::InputObject
description "File information required to prepare a direct upload"
argument :filename, String, "Original file name", required: true
argument :byte_size, Int, "File size (bytes)", required: true
argument :checksum, String, "MD5 file checksum as base64", required: true
argument :content_type, String, "File content type", required: true
end
argument :input, CreateDirectUploadInput, required: true
class DirectUpload < GraphQL::Schema::Object
description "Represents direct upload credentials"
field :url, String, "Upload URL", null: false
field :headers, String,
"HTTP request headers (JSON-encoded)",
null: false
field :blob_id, ID, "Created blob record ID", null: false
field :signed_blob_id, ID,
"Created blob record signed ID",
null: false
end
field :direct_upload, DirectUpload, null: false
def resolve(input:)
blob = ActiveStorage::Blob.create_before_direct_upload!(input.to_h)
{
direct_upload: {
url: blob.service_url_for_direct_upload,
# NOTE: we pass headers as JSON since they have no schema
headers: blob.service_headers_for_direct_upload.to_json,
blob_id: blob.id,
signed_blob_id: blob.signed_id
}
}
end
end
# add this mutation to your Mutation type
field :create_direct_upload, mutation: CreateDirectUploadNow, to retrieve a direct upload payload from the server, your GraphQL client must perform the following request:
mutation {
createDirectUpload(input: {
filename: "dev.to", # file name
contentType: "image/jpeg", # file content type
checksum: "Z3Yzc2Q5iA5eXIgeTJn", # checksum
byteSize: 2019 # size in bytes
}) {
directUpload {
signedBlobId
}
}
}…and some JavaScript
Disclaimer: the JS implementation below is just a sketch and hasn’t been tested in reality (since in my project we don’t use any of Rails’ JS code). All I checked for is that it compiles.
To upload a file, a client must perform the following steps:
- Obtain the file metadata (filename, size, content type and checksum)
- Request direct upload credentials and a blob ID via API—
createDirectUploadmutation - Upload the file using the credentials (no GraphQL involved, HTTP PUT request).
For step 1 and 3 we can re-use some of the code from the JS library that comes along with Rails (don’t forget to add "@rails/activestorage" to your package.json).
Let’s write a getFileMetadata function:
import { FileChecksum } from "@rails/activestorage/src/file_checksum";
function calculateChecksum(file) {
return new Promise((resolve, reject) => {
FileChecksum.create(file, (error, checksum) => {
if (error) {
reject(error);
return;
}
resolve(checksum);
});
});
}
export const getFileMetadata = (file) => {
return new Promise((resolve) => {
calculateChecksum(file).then((checksum) => {
resolve({
checksum,
filename: file.name,
content_type: file.type,
byte_size: file.size
});
});
});
};FileChecksum class is responsible for calculating the required checksum and is used by Active Storage in DirectUpload class.
Now you can use this function to build your GraphQL query payload:
// pseudo code
getFileMetadata(file).then((input) => {
return performQuery(
CREATE_DIRECT_UPLOAD_QUERY,
variables: { input }
);
});Now it is time to write a function to upload files directly to the storage service!
import { BlobUpload } from "@rails/activestorage/src/blob_upload";
export const directUpload = (url, headers, file) => {
const upload = new BlobUpload({ file, directUploadData: { url, headers } });
return new Promise((resolve, reject) => {
upload.create(error => {
if (error) {
reject(error);
} else {
resolve();
}
})
});
};Our complete client side code example would be the following:
getFileMetadata(file).then((input) => {
return performQuery(
CREATE_DIRECT_UPLOAD_QUERY,
variables: { input }
).then(({ directUpload: { url, headers, signedBlobId }) => {
return directUpload(url, JSON.parse(headers), file).then(() => {
// do smth with signedBlobId—our file has been uploaded!
});
});
});Looks like we did it! Hope that helps you to build your awesome new Rails + GraphQL project)
For a more realistic example, check out this snippet from our React Native application.
In Conclusion, or what to do with signedBlobId
Let me provide just a quick example of how we use signed blob IDs in the app—the attachProfileAvatar mutation:
class AttachProfileAvatar < GraphQL::Schema::Mutation
description <<~DESC
Update the current user's avatar
(by attaching a blob via signed ID)
DESC
argument :blob_id, String,
"Signed blob ID generated via `createDirectUpload` mutation",
required: true
field :user, Types::User, null: true
def resolve(blob_id:)
# Active Storage retrieves the blob data from DB
# using a signed_id and associates the blob with the attachment (avatar)
current_user.avatar.attach(blob_id)
{ user: current_user }
end
endThat’s it!
