American Fuzzy LopBlack Box Fuzzing

2016-10-24

• American Fuzzy Lop (AFL) is a fuzzing tool• Also, a breed of rabbit• Written by Michał Zalewski (`lcamtuf`)

• Used as part of a bunch of DEFCON presentations (including CGC)• Big in the computer security field

• Finds crashes in programs• Works on (instrumented) binaries• Has no understanding of problem domain

• Has found A LOT of tricky, impressive bugs:• http://lcamtuf.coredump.cx/afl/• Firefox, OpenSSL, clang, glibc, perl, screen, Redis…

American Fuzzy Lop

• Fuzz testing is the generation of random, hopefully invalid inputs

• Meant to catch the edge cases that you didn’t anticipate / test for

Aside: Fuzz Testing

1. Create a binary that reads from stdin, returns non-zero on exception2. Give AFL the binary and a few sample inputs3. Profit! (analyze crash logs)

Usage

• We want to check for crashes in the encode/decode functions of pyhocon

• HOCON (Human-Optimized Config Object Notation)• https://github.com/typesafehub/config/blob/master/HOCON.md

• pyhocon is a python library for HOCON SerDes• https://github.com/chimpler/pyhocon

Example

• We want to check for crashes in the encode/decode functions of pyhocon

1. Write a simple wrapper program

2. Since we’re in Python, we also use the python-afl library

3. Add the expected Exceptions

4. Run!

5. Wait…

Example

The code

It’s running!

• Bit + Byte flips• Arithmetic offsets• Troublesome values (0, 1, INT_MAX, etc)• Random overwrites + appends• Inserts/Deletes/Splices of inputs at random offsets

Checks

• Bit + Byte flips• Arithmetic offsets• Troublesome values (0, 1, INT_MAX, etc)• Random overwrites + appends• Inserts/Deletes/Splices of inputs at random offsets

Checks

• https://github.com/chimpler/pyhocon/issues/103• It happens when you try to append to an list that is nested in a dictionary

• Internally, a boolean was being passed in when it should have been a string

• While type checking would have also found this, a person manually testing likely would not (and did not) find it

Bug found!

• American Fuzzy Lop is a very good tool for black box fuzz testing of software.

• Very easy to use (nothing to learn, no domain knowledge)

• Especially useful for code:• That is complex• That you didn’t write• That you don’t have the source code for• ie. Code you don’t understand

• Further reading:• https://github.com/mirrorer/afl/blob/master/docs/technical_details.txt

• “10/10; would crash again”

Conclusions