Let’s Encrypt: TLS for NGINX

The following is adapted from a presentation given by Yan Zhu and Peter Eckersley from the Electronic Frontier Foundation (EFF) at nginx.conf 2015, held in San Francisco in September. You can watch the video of the complete talk on YouTube.

Table of Contents

0:00	Introduction
0:29	Problem #1 – TLS is Not Ubiquitous
2:10	Problem #2 – Setting up TLS
2:52	How Long Does it Take to Setup TLS?
6:30	Problem #3 – TLS Configuration is Confusing
7:01	RC4
7:58	SHA‑1
8:28	Logjam
8:48	SSL Test
9:06	Problem #4 – Mixed Content Blocking
11:06	W3C
11:45	Problem #5 – Too Many CAs
13:30	The Let’s Encrypt Mission
14:14	Let’s Encrypt
24:01	TLS and HTTPS Problem
25:30	Default Client Functionality
34:19	Let’s Encrypt NGINX
35:20	Parse Configs
35:47	Authenticator
36:28	Installer
37:28	Write Out the Config and Reload NGINX!
37:37	Let’s Encrypt NGINX Demo
39:33	Get Involved
40:13	Get in Touch

0:00 Introduction

Yan: Today, we’ll talk about a project called Let’s Encrypt and how we can use it with NGINX to provide TLS for the entire web.

I’m Yan Zhu. I’m a security engineer at Yahoo! and also a technology fellow at EFF.

Peter: I’m Peter Eckersley. I am Chief Computer Scientist at EFF.

00:29 Problem #1 – TLS is Not Ubiquitous

Yan: Let’s Encrypt helps solve a range of problems.

TLS is not ubiquitous even in 2015. For instance, when I joined a Wi‑Fi network a few minutes ago, I was able to go to Bing over plain HTTP and check whether there was a captive portal or not.

Here’s another example. In 2014, I was logging into Quora and I noticed that they were sending my password over in plaintext by default – unencrypted, no HTTPS. Also, they were trying to run Flash over HTTP, and I said no and I didn’t use Quora for a while after that.

Things have gotten better. Since then, Quora has implemented HTTPS by default, which is great. Many other sites have done that as well.

Even in 2015, companies as big as Google have pages, like this default landing page for Google Ads, that are still over HTTP.

You might say, “Okay, that’s not a problem because Google Ads is just public information. It’s just a picture and some information about ads”, right?

The problem is that a network attacker can inject the button that says Sign In and trick a user into clicking it, and maybe redirect them to a phishing site, and then scoop their passwords anyway. So, just having this one page, this one entry point, not be over HTTPS is a real risk.

2:10 Problem #2 – Setting Up TLS

The second problem is that setting up TLS is very tedious. How many people here have tried to set up HTTPS in the past? Quite a few, right? I actually was just doing it a few hours ago.

Usually, if you’re using a hosting provider like DreamHost, there’s this crazy‑long process.

You have to find some wiki page that tells you how to get SSL, you have to select their domain, generate a certificate signing request (CSR), make sure you copy and paste the CSR at the right place, and don’t accidentally copy and paste your private key. It all ends up being something like a 14‑step process.

2:52 How Long Does it Take to Set Up TLS?

We’ve done some usability testing and it takes developers, even experienced developers, at least an hour to set up TLS.

When I made this presentation, I actually went around EFF, found people who hadn’t ever set up TLS and asked them to try the process. One of them couldn’t figure it out. The other encountered lots of issues and end up taking nine hours to set everything up.

6:30 Problem #3 – TLS Configuration is Confusing

Let’s assume somehow we’ve managed to actually get ourselves an SSL certificate. Now, we want to actually configure our server to use TLS.

Then we run into the next problem, which is that TLS configuration is really confusing. It’s difficult to stay up‑to‑date on security best practices in order to keep your TLS configuration as secure as it should be.

7:01 RC4

In 2013 or 2014, it was still pretty common for people to use RC4 as a cipher algorithm. It was commonly believed that it was much faster than AES, so people would set it as a more highly preferred cipher than AES.

Here’s a blog post from Nick Sullivan, who gave an excellent talk earlier, that says it’s time to move on from RC4. That was in 2014 and now there’s even been more attacks against it. It’s just simply not good enough.

In fact, as of a few weeks ago, Google Chrome, Mozilla Firefox, and Internet Explorer, which I guess is now Edge, have deprecated RC4 entirely.

7:58 SHA‑1

Similarly, SHA‑1 is now being deprecated because collision attacks are too affordable now.

8:28 Logjam

Then there’s other attacks like Logjam, right? Now, you have to generate your own Diffie‑Hellman groups and all that.

Basically, TLS configuration best practices change fairly quickly. And if you’re not keeping up with that, then your configuration is going to be out of date and you’re going to have to go manually update your cipher suites all the time.

8:48 SSL Test

SSL Labs has a cool tool called SSL Test that you can use to see how well your SSL server configuration compares to current best practices.

The creator of SSL Test, Ivan [Ristić], also wrote a great book about SSL and TLS configuration which I recommend.

9:06 Problem #4 – Mixed Content Blocking

Let’s assume you’ve set up TLS and you have your configuration. You’re happy with it. Then you might run into a problem called mixed content blocking.

When you visit a page over HTTPS such as Lenovo.com, your browser does not want to load resources over HTTP after that. Essentially, if you’re on an HTTPS site and the URL bar says HTTPS, the user expectation is that they will be secure and they won’t have resources, such as scripts, loaded over plain HTTP.

So, the browser hard fails and basically says, “We’re not going to load any resources over HTTP if the site’s already HTTPS” – the top‑level site, that is.

What often happens when you switch over to HTTPS is you may have a lot of legacy resources that are still HTTP, and those resources simply get blocked. So, you end up with a site that is over HTTPS, but style sheets and images and scripts aren’t loading and your site looks pretty broken.

11:06 W3C

The W3C is also working on this problem because it’s preventing a lot of people from switching over to full HTTPS.

There’s now a content security policy directive called Upgrade‑Insecure‑Requests. Basically, your server sends back an HTTP response header that says upgrade‑insecure‑requests and the browser will try to fetch things over HTTPS instead of HTTP.

So, if your content supports HTTPS but you just don’t have the resources or don’t have the ability to go into all your HTML files and manually rewrite and add an “s” to [the scheme in] all the links, then you can use this header as a shortcut.

11:45 Problem #5 – Too Many CAs

The final problem is that there’s just a lot of certificate authorities [CAs], perhaps too many.

Peter did a study a few years ago in the decentralized SSL observatory and found that there’s a lot of intermediaries, right?

Peter: Yes. We scanned all of port 443 on the public IPv4 address space. And we downloaded all the certificates that we found there, and we tried to figure out who they’ve been issued by and whether browsers would trust them.

At the time everyone thought that there were about 50 or 60 certificate authorities that would be trusted by any browser, because that’s what you saw if you went and looked in Firefox or whatever browser you’re using.

But we discovered [that] in fact there were thousands of certificate authorities operated by probably hundreds of organizations that could actually issue a certificate for Google.com and be trusted by your browser.

The reason no one knew about them all is because they were being cross‑signed by other certificate authorities, and there was no central registry of who all these organizations were, but they are all part of the attack surface that’s applicable to basically any site on the web.

13:00 Maintaining Digital Certificate Security

Yan: So this is a real problem.

A few months ago, Chrome detected that there were unauthorized certificates issued for Google domains by an intermediate certificate authority which was issued by CNNIC.

13:30 The Let’s Encrypt Mission

Let’s Encrypt is going to try to solve all of those problems.

Peter: With Let’s Encrypt we are trying to offer two major solutions to the security problem. One is a solution for security, and that means turning on HTTPS and making sure that it actually provides the guarantees that it’s supposed to provide. And secondly, it needs to also be a solution for usability.

It’s no good to have really strong ciphers and really strong protocols if the humans who are supposed to use these things cannot succeed in doing so. And so, probably the biggest lesson we’ve learned in computer security in the last decade is that there is no security without usability. The two inherently have to come together or they don’t exist at all.

14:14 Let’s Encrypt

We are offering a new certificate authority called Let’s Encrypt.

Our solution to the problem of there being too many certificate authorities is to provide one more, but we’re trying to provide one more and actually do it right this time.

We’re offering really strong security guarantees, really great usability properties, and automation properties all the way through.

We’re taking an approach which we feel pretty confident is significantly better than what the current CAs are doing and should hopefully be a model that operates successfully at Internet scale. You can find out the details of how it works here.

24:01 TLS and HTTPS Problem

As Yan was describing, there are so many things that you need to get right in order to deploy TLS correctly.

Basically, you need a PhD in Computer Security or you need to be sitting around and reading news stories every week to follow the latest optimal configurations, so as not to be vulnerable to Heartbleed or Logjam or BEAST or CRIME – all of these attacks.

We don’t think everyone should have to know this stuff to stay secure, so we’re trying to provide a client or an agent that runs on your web server that basically does these things correctly for you in an automated fashion.

We’re trying to change the current situation where every web developer on the planet – and there are millions of web developers on the planet – is supposed to understand how TLS works in all its horrible, gory details. It’s kind of insane to expect every web developer to know about this stuff.

We’re trying to change that, so we can have a small team of people who study HTTPS and TLS configurations, figure out the best ones, and then get those automatically deployed to the millions of web developers who need to use HTTPS.

25:30 Default Client Functionality

The way the client works is it will tweak your web server. That can be NGINX or Apache or something else. It will tweak the server to pass the security challenges that are given to you by the certificate authority. Then it will install the resulting certificate, and then tweak all of your cipher suites and other security options so that you have a really good, secure setup.

And then it will automate the task of renewal so that you don’t have the problem (which anyone who’s deployed HTTPS will be familiar with) which is, you deploy it, and you go through hours of figuring out how to get your certificate. Eleven months pass, you’ve completely forgotten how you did it, and then your certificate expires, and your site goes down, and then you’re left back at square one again trying to remember what you did to get that certificate.

So, we’re trying going to try to automate both the renewal process and the response to certain kinds of security emergencies. In our ideal world, the next time there’s a Heartbleed‑type vulnerability, we’ll be able to do something, from people who’ve asked us to, in comparatively real time, to ensure that people are less vulnerable.

By the way, how many people here know about HSTS [HTTP Strict Transport Security]? Looks like a quarter of the room. That’s pretty good. If you don’t turn on HSTS for your website, your website is not secure. There is no such thing as a secure website that doesn’t use this thing. It is a necessary mechanism to ensure that when the user first types your domain name in the URL bar, they don’t get hijacked straight away and sent to some other phishing site.

HSTS also ensures that your cookies are structurally protected against trivially being hijacked. There are other ways to protect the cookies. There are no other ways to prevent stripping, but basically if you don’t know about HSTS, we’re going to need to help you find out about it and find out how to turn it on.

34:19 Let’s Encrypt NGINX

Yan: Now let’s talk about Let’s Encrypt on NGINX because that why we’re at the NGINX conference.

So, this is getting a little deep in the weeds of Let’s Encrypt development, but there’s a thing called a Let’s Encrypt plug‑in. We don’t have to merge these plug‑ins into our main code. Any of you guys can write plug‑ins, and the plug‑ins are usually custom implementations of one of these three interfaces, or some combination thereof.

An authenticator proves to the CA that you own some domain, an installer does the work of actually deploying the cert and configuring your SSL correctly, and the displayer replaces that really ugly ncurses UI, if you want to do that. And this is all done in Python. You can load them through setuptools.

Currently, we’re shipping with an authenticator and installer plug‑in for Apache and NGINX. I’m just going to talk about what the NGINX plug‑in (which I wrote) does.

35:20 Parse Configs

The first thing we do is we go through NGINX config files. You can specify the root directory if it’s not at C:NGINX, and we parse this into a Python data structure.

I say recursively because you can, for example, include configuration files in your NGINX main configuration. We’ll go through all of the included files, then we parse that into the sort of tree that we can now manipulate.

35:47 Authenticator

The authenticator does the work of doing authentication. As Peter said, one of the ways to prove that you own a domain is to be able to modify the server config files to set up a new virtual host. That’s what the DVSNI [Domain Validation with Server Name Indication] protocol does.

When we do the DVSNI proof, we actually have to mangle your config files a little bit to add a new server block that listens on a randomized subdomain. We do that, and then when we’re done, we reverse all those configuration changes. And hopefully that was successful, so now we can get the signed certificate from the CA.

36:28 Installer

Now, we have the certificate and we want to install that.

The plug‑in has some complicated logic to find the server block that best matches the provided hostname. The user experience is that the user says, “I want to get a certificate for example.com” and we’ll look through your config file and we’ll say, “What server block looks like it best matches example.com?” NGINX has a native way of doing this and we kind of mimic that.

When we find that block, we say, “Okay, now that’s the block we want to support SSL in”. We’ll have it listed on port 443 by default and add SSL to the listen directive, and then we’ll add these three lines at the bottom that say,

ssl_certificate your-certificate;

ssl_key your-key;

include ssl_options.conf;

ssl_options.conf is a file that we give you that has the current best practices for SSL – so the cipher suites and all that. You’ll probably tweak that depending on your server needs.

37:28 Write Out the Config and Reload NGINX!

Once that’s done, we just dump the whole thing back into your config file, reload NGINX and you should be good to go.

37:37 Let’s Encrypt NGINX Demo

So, what does this look like in practice? Here I have the server that’s running NGINX and I go to HTTPS, my subdomain, and it says not available – I’m sad.

Now I SSH in. This is just default NGINX on Ubuntu with TLS support, and that’s what my config file looks like. I think it’s just the default for NGINX.

So, here I’ve already installed Let’s Encrypt through my package manager or I’ve Git‑cloned it. So I [type] letsncrypt run, and I select NGINX because that’s what I’m using.

Then we get the ugly curses UI. If you don’t want to use that, there’s pure command‑line options you can do. Now, I select a domain or domains that I want to set up SSL for. Here it’s guessing which ones I have available by looking through my config files.

Now, I go back and I look at my config file.

Okay, and now you can see Let’s Encrypt has modified my server file to add these SSL blocks.

Right now when I go to the site using HTTPS it will just show a “connection not private” error, because we don’t have a valid signature from a CA yet. But in the future, that’ll just be SSL and you’ll get the nice screen‑lock icon and all that. And that’s it.

39:33 Get Involved

If you want to help, you can contribute on Github. For instance we’d love to hear if you have good ideas for config‑file parsing.

Peter: We definitely are interested in having a volunteer or contractor to help with the NGINX port in particular. We have it going for a lot of cases, but we want to make sure that it works for every NGINX configuration.

40:13 Get In Touch

Yan: Cool, that’s how you can get in touch with us at LetsEncrypt.org. I’m Yan, this is Peter, and that’s it. Thanks!

For more information about using Let’s Encrypt with NGINX, see Using Free SSL/TLS Certificates from Let’s Encrypt with NGINX on the NGINX blog.