Deploying Ruby apps to Google Cloud Kubernetes Engine continuously with CircleCI

May 1, 2018

Topics

Vladimir Dementyev
Principal Backend Engineer

I hadn’t had a chance to try Kubernetes for a long time, but finally a few weeks ago I got one: I’ve been working on a simple Ruby application (a Slack bot), and decided to deploy it to Google Cloud Kubernetes Engine.

There was an additional requirement to set up a continuous deployment process. For that, I chose CircleCI service.

NOTE: This post is just a playbook containing all the actions I made to get the application up and running.

Software versions:

Kubernetes 1.9.4-gke.1
Google Cloud SDK 195.0.0

Preparing GCloud

Let’s assume that you already have a Google Cloud account.

Go to the Console and create a new project (my-project) with Kubernetes API enabled (more thorough description of this step could be found here).

Next step—install gcloud CLI:

# for macOS/Homebrew users it's pretty simple
brew install caskroom/cask/google-cloud-sdk

Then you must authenticate yourself:

gcloud auth login

(Optionally) Configure the default project (to avoid specifying it with every command):

gcloud config set project my-project

Create K8S cluster:

# "g1-small" is a minimal instance type (as of the time of writing) that allows having only one node
gcloud container clusters create my-cluster --machine-type g1-small --num-nodes 1

Obtain the cluster’s credentials:

gcloud container clusters get-credentials my-cluster

Get the list of clusters to verify that everything is okay:

gcloud container clusters list

The output looks like this:

NAME         LOCATION    MASTER_VERSION  MASTER_IP      MACHINE_TYPE  NODE_VERSION  NUM_NODES  STATUS
my-app-web  us-east1-b  1.9.4-gke.1     35.227.92.102  g1-small      1.9.4-gke.1   1          RUNNING

Preparing application

As I’ve already told, the application I want to deploy is a simple Ruby-only app (i.e., no databases, caches, persistent stores). See the links at the end of the post on how to deploy Rails applications.

We need to build a Docker image and push to Google Container Registry.

Here is my Dockerfile:

FROM ruby:2.5.0

RUN apt-get update && apt-get install -y build-essential git

RUN mkdir -p /app
WORKDIR /app

ENV LANG C.UTF-8

COPY Gemfile Gemfile.lock ./
RUN gem install bundler && bundle install --jobs 20 --retry 5

COPY . .

# We want to include the current git revision information
# to a build to track releases
ARG BUILD_SHA=unknown

ENV BUILD_SHA ${BUILD_SHA}

EXPOSE 3000

ENTRYPOINT ["bundle", "exec"]

CMD ["puma", "-p", "3000"]

Let’s build it:

docker build -t my-app -f Dockerfile.prod .

NOTE: I use Dockerfile.prod file for production and Dockerfile for development.

Then tag it using a specific GCloud format (containing region and project name):

docker tag my-app us.gcr.io/my-project/my-app:v1

And push:

gcloud docker -- push us.gcr.io/my-project/my-app:v1

OK. We’re almost there. Now we have to tell somehow to our K8S cluster to get this image and run.

For that we’re going to use K8S deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      name: web
  template:
    metadata:
      labels:
        name: web
    spec:
      containers:
        - name: web
          image: gcr.io/my-project/my-app:latest
          ports:
            - containerPort: 3000
          livenessProbe:
            httpGet:
              path: /_health
              port: 3000
            initialDelaySeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /_health
              port: 3000
            initialDelaySeconds: 10
            timeoutSeconds: 1

K8S documentation is a good place to read about deployments; no need to recall it here.

I’d only like to pay attention to the livenessProbe and readinessProbe sections: they tell K8S how to monitor the application state to provide rolling update functionality and make sure that the desired number of replicas are up and running. Note that your application should provide endpoints for these checks (/_health in my case).

Now we need to install one more tool—kubectl—to operate our cluster:

gcloud components install kubectl

It’s time to finally deploy our application! Let’s create our deployment:

kubectl create -f kube/web-deployment.yml

To get the information about our deployment, you can see the list of pods:

kubectl get pods

More information about a pod:

kubectl describe pod web-<pod-id>

Updating application manually

Your application is up and running. How to push a new release?

If you want to update your deployment configuration, just run the command below:

kubectl apply -f kube/web-deployment.yml

For updating the codebase, you should tell your cluster to use a new image:

docker tag my-app us.gcr.io/my-project/my-app:v1.1
gcloud docker -- push us.gcr.io/my-project/my-app:v1.1
kubectl set image deployment web web=gcr.io/my-project/my-app:v1.1

K8S will take care of creating new pods and replacing the old ones.

Running three commands and manually providing new versions is not the most convenient way to deploy, isn’t it?

Let’s make someone else do all the dirty work.

CircleCI integration

CircleCI 2.0 is a very flexible automation tool.

We want to make our deployment as easy as just making git push (or merging PR into master branch).

Below you can find an example .circle/config.yml with the comments explaining every step:

version: 2
workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build
      - deploy:
          # Run the "deploy" job only if the "build" job was successful
          requires:
            - build
          # Run deploy job only on master branch
          filters:
            branches:
              only:
                - master
jobs:
  # This job is responsible for running tests and linters
  build:
    docker:
      - image: circleci/ruby:2.5.0
        environment:
          RACK_ENV: test
    steps:
      - checkout
      - run:
          name: Install gems
          command: bundle install
      - run:
          name: Run Rubocop and RSpec
          # Our default rake task contains :spec and :rubocop tasks
          command: bundle exec rake
      # Cache all the project files to re-use in the deploy job
      # (instead of pulling the repo again)
      - persist_to_workspace:
          root: .
          paths: .
  deploy:
    docker:
      # official image which includes `gcloud` and `kubectl` tools
      - image: google/cloud-sdk
        # project information
        environment:
          GOOGLE_PROJECT_ID: my-project
          GOOGLE_COMPUTE_ZONE: us-east1-b
          GOOGLE_CLUSTER_NAME: my-app
    steps:
      # Attach previously cached workspace
      - attach_workspace:
          at: .
      # SERVICE_KEY provides access to your GCloud project.
      # Read more here https://circleci.com/docs/2.0/google-auth/
      - run: echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
      # Authenticate gcloud
      - run: gcloud auth activate-service-account --key-file=${HOME}/gcloud-service-key.json
      # Configure gcloud (the same steps as you do on your local machine)
      - run: gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
      - run: gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
      - run: gcloud --quiet container clusters get-credentials ${GOOGLE_CLUSTER_NAME}
      # Enable remote Docker (https://circleci.com/docs/2.0/building-docker-images/)
      - setup_remote_docker
      # Build Docker image with the current git revision SHA
      - run: docker build -t brooder -f Dockerfile.prod --build-arg BUILD_SHA=${CIRCLE_SHA1} .
      # Use the same SHA as our image version
      - run: docker tag brooder gcr.io/my-project/my-app:${CIRCLE_SHA1}
      # Using remote Docker is a little bit tricky but this "spell" works
      - run: gcloud docker --docker-host=$DOCKER_HOST -- --tlsverify --tlscacert $DOCKER_CERT_PATH/ca.pem --tlscert $DOCKER_CERT_PATH/cert.pem --tlskey $DOCKER_CERT_PATH/key.pem push gcr.io/my-project/my-app:${CIRCLE_SHA1}
      # and finally, "deploy" the new image
      - run: kubectl set image deployment web web=gcr.io/my-project/my-app:${CIRCLE_SHA1} --record

Keeping secrets

Let’s cover one more topic here—managing app secrets.

We use Kubernetes Secrets to store sensitive information (such as third-party services API tokens).

First, you have to create secrets definition file:

# kube/app-secrets.yml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  # The value should be base64 encoded
  github_token: b2t0b2NhdA==
  slack_token: Y3V0bWVzb21lc2xhY2s=

Don’t forget to encode the value into base64. For example:

echo -n "myvalue" | base64

Push secrets to the cluster:

kubectl create -f kube/app-secrets.yml
#=> secret "mysecret" created

NOTE: Remove app-secrets.yml right after pushing to K8S (or, at least, add to .gitignore).

Now you can pass the secrets to your app through env variables:

# web-deployment.yml
apiVersion: apps/v1
kind: Deployment
# ...
spec:
  # ...
  templaee:
    # ...
    spec:
      containers:
        - name: web
          image: gcr.io/my-project/my-app:latest
          # ....
          env:
            - name: MY_GITHUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: mysecret
                  key: github_token
            - name: MY_SLACK_TOKEN
              valueFrom:
                secretKeyRef:
                  name: mysecret
                  key: slack_token