Revisiting Key Authentication Setups

Nearly 18 months ago I wrote about setting up and using key authentication to connect between hosts. I use it all the time and it's a major timesaver on all sorts of levels. When I recently set up a new server and moved a legacy stack I'd read an article talking up the benefits of the Ed25519 algorithm for keys...and given my extended vacation over the holidays I figured it was as good a time as any to do a review!

Why Does it Matter?

As with anything, over time clutter can accumulate and my own SSH keys were no exception. Also, aliases and keys of hosts of old were still hanging around in various places. Adding another server to the mix just seemed like a mess, so with a little extra time it was good to clean up stuff.

For the most part, I was already using RSA keys larger than 2048 bits, but I did find a few out there that were only 1024 bits long...a Really Bad Idea. Fortunately, with one exception, all of those "short" keys were on deprecated/decommissioned hosts, and the holdout was a very specific service. And all the digital clutter and shuffling around worked well to hide these...ostrich style!

Where to Begin?

I started out by looking at my most common endpoints (in this case, machines I usually work on/around). These already have Windows Subsystem for Linux (WSL) installed or are already some sort of Linux distro. I also have a mac to bring into the loop for a few things. I started out by perusing the known_hosts file in the ~/.ssh folder of my accounts and noted the places where I either wanted or needed to have key auth set up. From there I cross-referenced the authorized_keys in ~/.ssh to see where I'd already configured access and by what mechanism. For reference, each line in authorized_keys will have a format like this:

ssh-algorithm AAAAB3NREALLYSUPERLONGSTRINGOFVARIOUSCHARACTERSANDSOFORTH username@f.q.d.n

For example, this shows authorized RSA and Ed25519 keys for the same username@host combination:

ssh-rsa AAAAB3NREALLYSUPERLONGSTRINGOFSEEMINGLYRANDOMANDVARIOUSCHARACTERSANDSOFORTH username@f.q.d.n
ssh-ed25519 AAAAB3NLONGSTRINGOFVARIOUSCHARACTERSANDSOFORTHTHATSDIFFERENTFROMRSA username@f.q.d.n

Once I had a list of the common hosts and the keys, it was time to cull the herd.

So I Married an Axe Murderer?

Not quite, but it was easy to do a little hack-and-slash on the known_hosts and authorized_keys across all the endpoints and servers in a pass where I simply removed the lines and keys for any hosts no longer in service or used. This actually cleaned up a lot of stuff and made the rest far simpler to process and upgrade.

An Important Side Note: Old Host Support

The Ed25519 algorithm requires OpenSSH version 6.5 or newer. If for some reason (like me) you're still connecting to a stack using old OpenSSH (thanks, one old host--this happened to also be the one for which I had a "short" RSA key), note that so you don't remove your existing access. Ideally come back to that configuration and both upgrade the key (I changed mine to a 3072 length) and see if you can nudge the host owners to upgrade OpenSSH.

With a clean host list and notes as to where keys should be upgraded, it was time to generate some new stuff!

Generating the Ed25519 Key Pair

I used the recommended syntax from the Gist to create my new key pairs on my endpoints:

ssh-keygen -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "[my origin host name]"

I used the comment functionality to better identify if the original source machine/host, but that's not technically necessary. Future me will likely be thankful I did that. This is similar to setting up Github SSH keys, which is something you should totally do if you haven't already...especially if you work from the command line on server hosts!

Updating the Servers/Remote Hosts

Since I had the list of connection points from the previous step, I simply scp'd the new id_25519.pub key to each host. In most cases, this scp command used the older/existing RSA key pair I'd set up years before. In most cases I also appended the public key file name to include details about the new key (origin), e.g. id_25519-hostname.pub.

I'd then (on the remote hosts) jump over to ~/.ssh and add the new public key to authorized_keys:

cat ~/.ssh/id_25519-hostname.pub >> ~/.ssh/authorized_keys

I would then manually remove the "old" RSA key from authorized_keys and, if present, also remove the old id_rsa-hostname.pub file.

Exit the remote host, and give it another connection shot to see if it worked. Spoiler: it did!

ssh username@remotehost.name

Rinse and repeat across all the endpoints and remote hosts. Sure, it was a little time consuming and tedious, but it was glorious to have all the old crap cleaned up!

While I Was There...

I'd had a few specific key setups in the ~/.ssh/config file before, namely for specifying the correct key for Github and Github Enterprise in a connection, but also a couple others. While I was at it, it seemed a good and reasonable action to reset and standardize this across endpoints...most specifically the WSL instance where my WSL account is a different name. In the end, I set up config entries for some hosts along this line:

Host github-enterprise
  User zaskem
  HostName github.enterprise.name
  IdentityFile ~/.ssh/id_github_enterprise
Host hostname
  User zaskem
  HostName other.host.name
  IdentityFile ~/.ssh/id_ed25519
Host alternate-hostname
  User service
  HostName other.host.name
  IdentityFile ~/.ssh/id_ed25519-servicekey

In most cases this isn't necessary as SSH will figure out the correct key to use, but it's a way to explicitly specify the key if you have multiple accounts and specific service needs (I have some service accounts that use their own key pair). The config file is a convenient way to address any situations where you might otherwise need to specify the key/identity in a connection, like so:

ssh -i ~/.ssh/id_ed25519-servicekey service@other.host.name

With the config definition becomes:

ssh alternate-hostname

Life: Cleaned Up and Improved

It's good to periodically review your old configurations like this and cull the herd or make improvements for other nuisance situations (see the -i argument in the example above). Of course, actually carving out the time and energy to do so is easier said than done. But at the end of the day this entire process didn't take very long, and I'm very happy with the new key setup and consistent application of configurations, aliases, etc.

Well worth the effort, especially if you spend any time jumping around between hosts!