Stick with security: YubiKey, SSH, GnuPG, macOS

Evil Martians are growing. With more employees and more clients, there is a demand for stronger security. Our clients trust us with their source code and, even more importantly, with access to their production servers, and this trust cannot be broken. In a hostile environment of the modern web, though, it is easier said than done. A good old password, even coupled with a password manager, does not cut it anymore. The most obvious way to increase security is to opt for two-factor authentication (2FA) that is widely supported. Even without hardware keys, it makes an attacker’s job much harder than it used to be.

A sticky situation

We have enforced 2FA across all our staff for all the tools that we use daily: email, GitHub, task trackers, and others. By default, it involves requesting one-time access codes either by SMS/phone call or through a dedicated smartphone app. Cellular networks, however, are not the safest place: messages and calls can be intercepted. Opting for an app like Google Authenticator is more secure, but can also be compromised, at least in theory, if a smartphone that runs it is precisely targeted by an attacker.

So, can we do better? There exists an open authentication standard that aims to both strengthen and simplify 2FA.

Known as Universal 2nd Factor (U2F) and originally developed by Yubico and Google, it relies on physical devices (usually USB or NFC) that implement cryptographic algorithms on a chip, similar to smart cards that have been around for ages. You probably have at least few of those in your pockets: phone SIM, bank cards, various IDs and the like.

Now, instead of confirming your access with some code, you need to insert a USB stick into your computer, press the physical button on it, and the device will take care of the rest. Authenticating with U2F is already supported by major browsers (the only notable exception, sadly, is Safari) and you can use it with many online services that software professionals use daily: Google and Gmail, Dropbox, GitHub, GitLab, Bitbucket, Nextcloud, Facebook, and the list goes on.

The advantages of a hardware solution are obvious: a possibility of a remote attacker gaining access to one of your tools is pretty much eliminated. The attacker needs to physically get a hold of your USB key, which is still a security risk, but in an entirely different domain.

There is a number of vendors that sell USB keys, and we chose Yubico and their YubiKey 4 series. They are versatile, compact and can either be carried around on a keychain or, for smaller models, stay in the USB slot of your laptop all the time. There are also USB-C models for newer Macs, so you don’t need dongles. Besides implementing U2F, YubiKey 4 series supports various security standards:

• Yubico OTP
• Smart card PIV
• OpenPGP
• OATH-TOTP (Time-based)
• OATH-HOTP (HMAC-based)
• Challenge-Response

Authenticating online with U2F works out of the box on Linux, macOS, and Windows and in all major browsers. However, if you want to use your YubiKey for SSH connections, things quickly get less straightforward.

Sticks and Macs

A Mac is a computer of choice for most of us at Evil Martians. We also use SSH all the time: while pushing code to GitHub or accessing remote servers. As all our employees work remotely from their private machines, contents of their ~/.ssh folders should never be allowed to leak. Common security measures, like the hard drive encryption, are always in order, but with YubiKeys already being used for U2F, would not it be better to store RSA keys for SSH on them too, and off the computer?

As YubiKey already supports OpenPGP, we can use it as the OpenPGP card with all the benefits:

• Once RSA keys are put on a card, they cannot be retrieved programmatically in any way.
• Keys written to a card can only be used in combination with a PIN code, so even if a YubiKey is stolen, a thief would not be able to authenticate directly.

To set up YubiKey as a smart-card holding your PGP keys, you need first to replace your ssh-agent that comes pre-installed with macOS with a GnuPG solution. The easiest way to do it is directly from Terminal with Homebrew:

$ brew install gnupg

If you want to install a full GPG Suite that includes GUI applications, you can run another command (requires Homebrew Cask), or download it from the website:

$ brew cask install gpg-suite

At the time of this writing, the most recent version of gpg is 2.2.X. Let’s double-check, just to be sure:

$ gpg --version
gpg (GnuPG) 2.2.6
...

Important! Now you need to either generate your PGP keys directly on the YubiKey or create them locally and copy over. There is an official guide for that, as well as a more evolved instruction on GitHub from the user drduh.

After all that is done, you need to enable your SSH client (the built-in Terminal app, for instance) to read PGP keys directly from YubiKey. It is time to say goodbye to a built-in ssh-agent that have served you well before.

Insert a YubiKey holding a PGP key in your computer and run the following commands; they will launch a gpg-agent and instruct your applications to use a new SSH authentication socket:

$ gpgconf --launch gpg-agent
$ export SSH_AUTH_SOCK=$HOME/.gnupg/S.gpg-agent.ssh
$ ssh-add -l

If everything went well, you should see that your private RSA key is now in fact located on a YubiKey (it has a unique cardno), the output of an ssh-add -l should resemble this:

2048 SHA256:XXXXXXXXXXXXX cardno:000XXXXXXXXXX (RSA)

Congratulations, you are done! This changes will not persist, however.

As soon as you reload your system, or even switch to a new console window, this setup will go away.

Let’s see how we can make it permanent.

Making things stick

The first thing that comes to mind when changing any shell-related setup is to change the local profile, be it ~/.bash_profile or ~/.zsh_profile (if you don’t know what type of shell you have, most likely you have bash, it comes by default with macOS). Open that file in an editor and add:

gpgconf --launch gpg-agent
export SSH_AUTH_SOCK=$HOME/.gnupg/S.gpg-agent.ssh

Now every time you launch a console, it will know how to SSH properly. If you live in a shell, use Vim or Emacs to write your code and were never tempted with GitHub’s visual features, you are all set.

However, if you use an IDE or one of those modern text editors with integrated GitHub functionality, such as Atom or Visual Studio Code? Those applications are not concerned with your shell setup and will still use system defaults for SSH, which is not what we want since we store all our keys securely on a YubiKey.

“But before we dealt with gpg, we did not need to set up anything, and everything worked!” you might say, and you would be right: macOS takes care of all that with a built-in service-management framework called launchd. You can read more about it by running man launchd, but you don’t have to.

You only need to know that launchd deals with so-called “property lists”. These are XML files with a .plist extension that define services to be launched and their launch options. They are located in the following directories:

• ~/Library/LaunchAgents for per-user agents provided by the user.
• /Library/LaunchAgents for per-user agents provided by the administrator.
• /Library/LaunchDaemons for system-wide daemons provided by the administrator.
• /System/Library/LaunchAgents for per-user agents provided by macOS.
• /System/Library/LaunchDaemons for system-wide daemons provided by macOS.

Let’s do some digging and look for anything SSH-related. Here it is, right in a /System/Library/LaunchAgents/com.openssh.ssh-agent.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.openssh.ssh-agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/ssh-agent</string>
        <string>-l</string>
    </array>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
        <dict>
            <key>SecureSocketWithKey</key>
            <string>SSH_AUTH_SOCK</string>
        </dict>
    </dict>
    <key>EnableTransactions</key>
    <true/>
</dict>
</plist>

Without diving into much detail, we see that this is how macOS makes ssh-agent a default utility for SSH authentication.

Unfortunately, we cannot edit this file directly, as anything located in a /System folder is protected from tampering by a macOS feature called System Integrity Protection. There is a way to disable it, but you don’t want to do that. Apple folk came up with it for a reason.

A stickler for detail

Nothing prevents us from writing our own .plist though! All these XMLs will be treated as instructions for launchd, so this is our chance to circumvent ssh-agent once and for all!

First of all, let’s read man gpg-agent and learn what GnuPG agent for Mac is capable of:

• It can become a daemon and live in the background with the --daemon option.
• There is a --supervised option designed for systemd which makes the gpg-agent wait for a certain set of sockets and then access them through file descriptors.
• A --server option allows our agent to hook onto the TTY and listen for text input, without opening any sockets.

Unfortunately, launchd only tracks processes that run in the foreground, and neither --supervised, nor --server will do us any good. So, the best way to launch an agent is by using the same command that we used before: gpgconf --launch gpg-agent. Let’s express it in launchd-compatible XML:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <!-- Sets a name for a task -->
  <key>Label</key>
  <string>homebrew.gpg.gpg-agent</string>
  <!-- Sets a command to run and its options -->
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/gpgconf</string>
    <string>--launch</string>
    <string>gpg-agent</string>
  </array>
  <!-- Tells it to run the task once the XML is loaded -->
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Now save it as homebrew.gpg.gpg-agent.plist and put it into ~/Library/LaunchAgents folder. To test that it all works (you won’t have to do it after restart), tell launchd to load a new plist:

$ launchctl load -F ~/Library/LaunchAgents/homebrew.gpg.gpg-agent.plist

Now let’s make sure that the agent is loaded:

$ launchctl list | grep gpg-agent
-   0   homebrew.gpg.gpg-agent
$ pgrep -fl gpg-agent
33399 gpg-agent --homedir /Users/example/.gnupg --use-standard-socket --daemon

A digit in the launchctl list output shows the exit status of a launched program, and 0 is what we want to see. pgrep confirms that we are in fact up and running.

However, we are not done yet. We still need to point SSH_AUTH_SOCK environment variable to $HOME/.gnupg/S.gpg-agent.ssh. The problem is that the variable is already set (user-wide) by the launchd default setting for ssh-agent.

At this moment, I have nothing better in mind than the following “hack”: forcibly symlink gpg-agent’s socket to the default one, stored in an SSH_AUTH_SOCK variable. The power of Unix allows us to do that, but that effectively messes up the default SSH configuration. However, as we are now using GnuPG for everything SSH-related, that should not be a problem. If you have better ideas, please contact me on Twitter.

We can create another plist that will do all necessary symlinking on login.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>link-ssh-auth-sock</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/sh</string>
    <string>-c</string>
    <string>/bin/ln -sf $HOME/.gnupg/S.gpg-agent.ssh $SSH_AUTH_SOCK</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

The only trick here is to call the shell directly, with /bin/sh (so we can reference shell variables), and pass a command to it. Now, save the file as ~/Library/LaunchAgents/link-ssh-auth-sock.plist and load it with launchd.

$ launchctl load -F ~/Library/LaunchAgents/link-ssh-auth-sock.plist

Let’s test the result:

$ launchctl list | grep link-ssh-auth-sock
-   0   link-ssh-auth-sock
$ ls -lah $SSH_AUTH_SOCK
lrwxr-xr-x 1 example wheel 36 Mar 22 19:10 /private/tmp/com.apple.launchd.XXXXXXX/Listeners -> /Users/example/.gnupg/S.gpg-agent.ssh

Bingo! Our macOS is now effectively tricked into thinking that it deals with ssh-agent, even though it’s the gpg-agent doing authenticating and reading PGP keys directly from your YubiKey.

All you need to do know to authenticate over SSH in a true hardware fashion is to turn on your laptop, put a stick in the USB and push a button on it. Your Mac is now completely secure!

Still stuck?

There is another problem you may encounter when you start using YubiKey as an OpenGPG card. Our gpg-agent sometimes get stuck, and it looks like a YubiKey is not connected at all, replugging it also does nothing. It is a known problem, discussed here.
My observations show that it appears after I put my laptop to sleep. Let’s deal with that too.

First, we need a tool that keeps track when our laptop wakes up: sleepwatcher is made just for that. Install it with Homebrew:

$ brew install sleepwatcher

By default, it expects two scripts: ~/.sleep to run before the computer goes to sleep, and ~/.wakeup to run after it wakes up. Let’s create them.

The minimal ~/.sleep script can look like this (we only need to be sure it passes as a shell script)

#!/bin/bash

exit 0

In ~/.wakeup we will forcibly restart our gpg-agent:

#!/bin/bash

/usr/bin/pkill -9 gpg-agent
/usr/local/bin/gpgconf --launch gpg-agent

exit $?

Now we need to add execution flags and enable sleepwatcher’s service:

$ chmod a+x ~/.sleep ~/.wakeup
$ brew services start sleepwatcher

Thank you for reading! In this article, we showed how to set up your SSH authentication flow with YubiKey as an OpenPGP card and how to make your gpg-agent play nicely with macOS. Now all you need to do to access a server or push code to a remote repository is to insert a stick into your USB and enter a PIN code when requested. Passphrases no longer required!