Removing Staples: Moving Away From Let's Encrypt OSCP Stapling

Last summer, Let's Encrypt announced their intent to end their OCSP service, and this was formalized in December with key dates related to the change. Near the end of January, I also received a direct email from Let's Encrypt detailing certificates of mine that had been configured with the "Must Staple" property. In my case, I had about 15 certificates whose ACME configurations requiring OCSP stapling, so it was time to make the change so I wouldn't have anything fail to renew come May 7.

The Basic Configuration

The vast majority of my footprint in this case is Linux and Apache (with Certbot as my ACME client). That means there are ultimately two places to make this change in order for stapling to be removed:

1. The Certbot certificate renewal profile; and
2. The Apache SSL configuration.

Not all of my domains/things/certificates had been configured for stapling since I'd added the SSLUseStapling On directive in the affected Apache site configurations (instead of globally or via an include). Your mileage may vary, but ultimately you need to find the place(s) where this might have been set.

A Nuance To The Change

If you simply update the Certbot/ACME client and Apache configurations, Chromium browsers will quietly ignore the interim configuration (a certificate that requires stapling and a server that isn't providing it), but Firefox (and presumably other clients) will throw an error about the configuration change if the certificate isn't also cycled/replaced in the process:

 This error is not bypassable in my very limited testing, so for best results you will want to renew the certificate at the same time you make the stapling change.

The [Simple] Process

I used this order when making the change for each affected certificate:

1. Update the Apache site configuration to remove the SSLUseStapling On directive (alternatively it could be set to Off).
2. [Optional] Restart Apache.
3. Update the Certbot renewal profile for the certificate.
4. Force renew the certificate.
5. [Optional] Check/Verify the configuration with the SSL Server Test service.

I have two optional steps here. Technically Certbot will restart Apache when certificate renewal is successful, but I did a restart on my own just to be safe. Additionally, I just wanted to verify with the SSL Server Test service that I hadn't screwed anything up along the way. Your mileage may vary.

Step 1: Update Apache Configuration

In the "sites" I had configured for OCSP stapling, there were two directives to remove/comment out (e.g. /etc/apache2/sites-available/name-of-site-ssl.conf):

SSLStaplingCache shmcb:/var/run/apache2/stapling_cache(128000)
SSLUseStapling On

Without stapling enabled, there's no need to specify the cache, so it's safe to remove as well.

Step 2: Restart Apache

Pretty straightforward. If you're unsure, though, running the configtest can help identify any typos or errors before they kill Apache on restart:

$ sudo apachectl configtest
Syntax OK
$ sudo apachectl restart

Step 3: Update Certbot Renewal Profile

With Certbot, the default path for certificate renewal profiles is generally /etc/letsencrypt/renewal. Look for the must_staple option in the affected profile and set it to False:

# Options used in the renewal process
[renewalparams]
must_staple = False

Step 4: Force Renew the Certificate

For this step you'll need to identify the certificate name, which can be identified from the output of the certbot certificates command. In this case, I'm going to --force-renew the --cert-name zaskecode.com certificate:

$ sudo certbot renew --cert-name zaskecode.com --force-renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/zaskecode.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for zaskecode.com and www.zaskecode.com
Reloading apache server after certificate renewal
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded:
  /etc/letsencrypt/live/zaskecode.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Step 5: Verify!

Verification can be as simple as just using browsers or clients, but I like the SSL Server Test service when I make "changes" to site configurations like this. The output is super useful to examine your configuration and goes quite in depth. In particular, related to this change, you will want to look for the OCSP stapling detail of the report. It should indicate No and no other errors should be displayed in the output:

 The output of the report can also point at things you may want to consider with your configuration. In this case, my results were all with an "A+" overall rating (see headline photo above)!

It's A Process, But Manageable!

As I noted, I had about 15 of these to change and it ultimately took about 25-30 minutes to get them all switched over and renewed. Of course, doing all the conversions at the same time helped, but you can do it too!

Certificates don't need to (and shouldn't be) scary or complicated. Use the tooling available and you should also have success! Hopefully this short set of instructions is useful -- good luck!