Fuzzing nginx - Hunting vulnerabilities with afl-fuzz

No 0day here

If you were looking for it, sorry. As of 48 hours of fuzzing, I’ve got 0 crashes.

AFL - successful fuzzing

American Fuzzy Lop has a very impressive history of finding vulnerabilities. The trophy case is gigantic. An ELI5 of the design of the product is: Give it a program a valid input file, and it will mess with that input file until using it crashes the example program. My first attempt at using it almost immediately found a crash situation in lci - Lolcode interpreter.

Unfortunately, successful use against something which is not a command line application that runs and quits is more difficult.

Compile and build

Our first step here will be to compile afl. I’m going to assume you can already do this. When building nginx, I used the following commands:

export CC=/path/afl-clang
./configure --prefix=/path/nginxinstall --with-select_module

The use of the prefix is simple - we don’t want to install this as root, as a proper service, or run it as such. The select module, I’ll get back to. With nginx built and installed, there are some very helpful config options:

master_process off;
daemon off;
events {
    worker_connections  1024;
use select;
multi_accept off;

}

By starting your config file like this, nginx will helpfully avoid forking to background, and start itself at a console where it belongs.

Your first server section should look like this:

server {
    listen       <ip>:8020;
    ...
}

We do this because:

* We want the parser to decide it's happy to run as non-root
* Without specifying the IP, something doesn't bind properly in our later process.

Operate with stdin/stdout

Following the suggested build gets you halfway there, but the remaining problem is that nginx wants to take input from a network port, not from stdin. Fortunately, this project exists:

Preeny on Github

Preeny almost solves our issues. I say almost because of two things:

  • Preeny intercepts accept(), but, where it exists (my system), nginx uses accept4()
  • nginx’s default polling mechanism simply doesn’t recognise connections that have been redirected and never triggers the event loop

For the first of these, I wrote this patch. Given accept() and accept4() are equivalent enough for our purposes, this patch just pushes accept4() to the intercepted accept().

diff --git a/src/desock.c b/src/desock.c
index 36b3db7..4b267ef 100644
--- a/src/desock.c
+++ b/src/desock.c
@@ -209,6 +209,11 @@ int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen)
        else return original_accept(sockfd, addr, addrlen);
 }

+int accept4(int sockfd, struct sockaddr *addr, socklen_t *addrlen, int flags)
+{
+       accept(sockfd, addr, addrlen);
+}
+
 int bind(int sockfd, const struct sockaddr *addr, socklen_t addrlen)
 {
        if (preeny_socket_threads_to_front[sockfd])

Again, compile as per the Preeny instructions, I won’t walk you through this.

Running it

With this in place, you can run nginx from the command line, and have it take HTTP syntax from stdin.

$ LD_PRELOAD="/home/technion/attack/preeny/Linux_x86_64/desock.so "  ./nginx
--- Emulating bind on port 8020
GET / HTTP/1.0

HTTP/1.1 200 OK
Server: nginx/1.8.0
Date: Tue, 28 Apr 2015 09:18:51 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 27 Apr 2015 08:45:32 GMT
Connection: close
ETag: "553df72c-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
	            width: 35em;
			           margin: 0 auto;
					           font-family: Tahoma, Verdana, Arial, sans-serif;
						       }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

This is successful.. almost. The problem you now see is that nginx never actually exits. To get around this, we had to patch nginx itself. Specifically, at line 262, I added this:

static int first_fd = 0;
    if (first_fd == 0)
            first_fd = max_fd;

    if(max_fd > first_fd) {
            printf("Exiting cleanly\n");
            exit(0);
    }

I’m sure there’s a better place to patch, but this seemed to be the easiest for me to find. Specifically, when it knows it’s been through the event loop once before and actually accepted a connection already, it’ll log as such and exit.

Now, let’s get a proper test case up and running. I created testcases/in.txt, based on a standard HTTP connection:

GET / HTTP/1.1
Acceptx: text/html, application/xhtml+xml, */*
Accept-Language:en-AU
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Host: lolware.net
DNT: 1
Connection: Keep-Alive
Cookie: A=regregergeg

Now let’s execute it and see how that looks:

$ LD_PRELOAD="/patch/preeny/Linux_x86_64/desock.so "  ./nginx < testcases/in.txt
--- Emulating bind on port 8020
HTTP/1.1 200 OK
Server: nginx/1.8.0
Date: Tue, 28 Apr 2015 09:43:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 27 Apr 2015 08:45:32 GMT
Connection: keep-alive
ETag: "553df72c-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Exiting cleanly
$

That right there is perfect. It takes the input file from stdin, and passes it to nginx, outputs the HTML web content, then quits.

Now all that’s neccessary is to run it under afl-fuzz:

$ LD_PRELOAD="/home/technion/attack/preeny/Linux_x86_64/desock.so " /home/technion/afl-1.61b/afl-fuzz -i testcases -o findings ./nginx

Now hang on, this’ll run for a while.

nginx - Built against LibreSSL

CentOS

For some time, I’ve been managing a CentOS RPM of LibreSSL built against nginx. You can still get that at the below link if you’re interested, but as of April 2015, I’ve moved to Arch as my preferred OS.

Nginx LibreSSL RPM Source

Nginx built against LibreSSL

I don’t currently recommend, unless you are running OpenBSD, using LibreSSL. There are too many untested applications. Testing nginx, is something I wanted to take on.

Regardless of whether you want to use by build or anything else, the fact remains: This page used to contain a set of instructions regarding how to patch up nginx and get it running with LibreSSL. At the present time, due both to smarter integration on the nginx side, and compatibility patches I’ve submitted to LibreSSL, things currently “just work”.

Linking nginx against LibreSSL gives you a very reliable method of implementing Chacha20/Poly1305 cipher in nginx. I’ve been using this string:

    ssl_ciphers "ECDHE-RSA-CHACHA20-POLY1305 ECDHE-ECDSA-CHACHA20-POLY1305 EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA  !aNULL !eNULL !LOW ! 3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

This gives A+ on the SSL labs test, and negotiates with Chacha20 when possible.

The move to Arch

A reasonably high component of my contribution to the open source community has related to identifying compatibility issues with current versions of CentOS. I didn’t ask for that. I just wanted to try these applications, and found I couldn’t. After spending a solid three hours with oclint, I’d had enough and made a platform move.

One of the great things about this is I can submit my build to the AUR without it being a big deal.

Get it here

Taking the Matasano Crypto Challenge

Salespeople

Yes, I’m putting this first. Even before introduction. If you’re in sales, the moment you see something slightly technical you’ll tune out, and I want you here.The Matasano Crypto challenge has some interesting elements for you to consider too. You probably won’t want to even attempt it, but there’s something you can learn from it anyway. Think about what you tell me when you want to sell security. You invariably use the letters “AES”, you tell me it’s invincible and you tell me that’s all the technical details that matter. I’ve probably heard this from three different vendors in the last few weeks. Now let me draw your attention to challenges 12, 13, 14, 16, 17 and 20. Every one of these challenges a person to actually implement a different, practical method of cracking AES. And before someone chimes in telling me none of these attacks would work in the real world, a quick search on Github pointed me at three different vulnerabilities I could easily exploit. So if you want to talk to me about selling security, you better have something better than the letters AES.

Introduction

As per the title, I’ve worked my way through the Matasano Crypto Challenge. When this was announced last year, I was pretty excited by the description. Here’s why. Allow me to point you at a a well respected cryptanalysis paper. Bored yet? Me too. I would say “you’d need a math degree to make sense of it”, but I majored in maths and still can’t follow. So when an opportunity comes to write a practical cryptographic attack, it’s interesting. I’d like to extend an absolutely huge thanks to the Matasano crew for putting this together. Now that the rules have been lifted (I never managed to get my hands on it before it went public) I’d like to write up some “tips”, for those interested in help without taking a major spoiler. Of course, if you just want code, complete solutions to the Matasano Crypto Challenge can be found here.

Some notes on Ruby

This exercise served a dual purpose - I used it to learn Ruby as much as study cryptography. When I hit challenge 1, I couldn’t write hello world. These challenge is an absolutely amazing way to learn a language. When I started this, I just didn’t accept anything other than C would manage. When you hit the later exercises, I shudder to think how long that would have taken me in C. Even things like actually working with (and cracking!) 768 bit RSA was trivial on my entry-level Linode. The language really is incredibly powerful. One thing about ruby- get used to writing this a lot: .force_encoding(“ascii-8bit”). Plenty of functions worked great for one input, and died with encoding errors on another.

Set one

The temptation here is to say “this is easy, I’ll skip it”. You’re only going to hurt yourself. Everything here is going to be cut and pasted into a later challenge anyway, so bite the bullet and do it. I learned Ruby throughout these exercises (it probably shows, my set 1 code is far more terrible than the latter), and I’d encourage you to use the opportunity similarly.

Challenge 6: Look at the wording on point four. It suggests averaging the result across four blocks. I don’t know if it was just me, but four blocks pointed firmly at an incorrect keysize. I struggled with this for a while until I average much more blocks, for a much larger keysize.

Set two

Challenge 9: I thought I completed this correctly, until later code which used this implementation broke. What’s not described here is that the PKCS #7 standard defined what to do when input is exactly a blocksize in length. Instead of naively appending 0 null bytes (how would that even work), append an entire block fo BLOCKSIZE.

Set three

Challenge 23 was an utter pain in the ass. It’s the one challenge I somewhat thre in the towel on and ported someone elses code for. If anyone can point me at a sensible description of the maths, I’d love to see it.

Set four

  • Challenge 29: Keep your solution lying around. Seriously, sooooo much is vulnerable to this.
  • Challenge 31: Even with the artificial delay cranked up to one and a half seconds, I couldn’t reliably determine the key. The protip here is: Webrick is terrible. After changing to Unicorn, the same peice of code was effective down to about 300ms. That’s still higher than the suggested 50ms, and I could see how to improve the code, but that’s already the next exercise.
  • Challenge 32: It’s amazing how accurate you can get this using not just an average, but an average that trims outliers, of which there always seems to be a few. My code was able to determine the key even down to 2ms.

Set Five

Things get a lot slower here. That said, although it’s a lot more work, don’t let the warnings scare you. I wouldn’t personally call this set any harder than set three, although I can see how people drop out due to the time investment. For an invmod function, check rosettacode.org.

Challenge 33: This algorithm is interesting to implement in Ruby. Variable names in the algorithm are “a” and “A”. Except trying to use that in Ruby gives you a big fat warning about assigning a constant. Sure, you can just use a different letter, but it’s surprising how mentally draining it gets reading one letter and writing another every time.

Challenge 38: This is a good demonstrating in understanding why certain algorithms do certain things. Having B depend on the password is done for a good reason.

Set Six

  • Challenge 42: I thought I was being smart utilising SHA256 as the hash in this challenge, where every write up used SHA1. Turns out, not only is SHA256’s digest longer, the ASN identifier is longer. The end result is that there is barely any room for trash at the end and as far as I can see, this won’t work. That cost me a few days. The other interesting thing is, for every “string to integer” algorithm I could put together (and observed in other people’s answers) that leading null byte dissapears. Todo: What’s the proper solution here? Finally, the cube root. Turns out it’s not trivial to do this using big integers. The suggestion here is, find an “nthroot” implementation in your language. In Ruby, it will try to use floats, which will give you “Infinity” answers. Forcing integers loses precision, but perfectly suits this use case.
  • Challenge 43: See that string, the hash and the integer they get from it? There’s a catch there. That string has TWO different \n characters. One on the line break you see, and one on the end. Yep, that had me questioning whether I borked SHA1 in every previous challenge for a few hours.
  • Challenge 47: Boy they weren’t kidding about this being a lot harder. If you want to peek at solutions, there are a total of two of them I found on Github. There’s one python solution where the writer actually implemented challenge 48 and then commented that parts of his code just weren’t needed until the next challenge. Also, it wouldn’t execute on my machine and I didn’t want to get bogged down in Python to investigate. In short, that didn’t help. The second solution I found was done in Java, which, to me, is unreadable. In short, you really are on your own. Where this is exceedingly hard is that there’s no way to check the intermediate steps. If your “step2a” is broken, the rest of your code just won’t work. Again, pay attention to wording. “Probably not going to need to handle multiple ranges”. With a working solution in place, half the time my solution cracks a plaintext correctly, and half the time it runs into multiple intervals and produces garbage. In short, where your range of ‘r’ is greater than one, at least raise an exception so you don’t go on a bug hunt. Secondly, I don’t understand step 4. The actual paper provides it as part of the solution, but you have a working, cracked plaintext without it. Of the other two solutions I’ve reviewed, one of them implements it, one doesn’t, both claim to solve the problem equivalently, leading to much confusion.

Set Seven

  • Challenge 49: I had absoutely no idea that anyone anywhere was using CBC-MAC. It can be shocking what you’ll find on Github.
  • Challenge 50: See that “extra credit”. Let’s just say the list of “things I would rather do than any challenge starting with ‘Write Javascript’” is an exceedingly large list. That said, if anyone else implements it, I’d be interested in using it to test my solution.

Closing

Things I don’t pretend to be good at: Front end web dev. This page is terrible. I know it. I’d rather continue to get better at the backend and let the creative guys do what they do. Seriously though, everyone should try these challenges. If you’re an entry level dev and you think it’s over your head - good. Maybe you won’t try and argue with someone about encryption implementations. If you’re a crypto genious, how about taking a break from the maths and writing some code. And yes, literally hours after writing this, set 7 got released. Obviously, I haven’t done it yet.