Using AFL to fuzz ARM binaries on a Rasbperry Pi and with AFL QEMU mode

afl-fuzz running on a Raspberry Pi B

I am currently taking a closer look at a certain range of I²C devices that promise an easy way to enhance embedded systems security. My main goals here are to check the SDK and drivers, which are partially closed source, for potential security problems and maybe to reverse engineer the proprietary protocol that is used to talk to the devices. Since these devices are not yet publicly available at the time of writing I can not yet publish any real world code. That also means that the screenshots and examples you are going to see are not based on production code but were made to resemble the real thing as closely as possible.
Another consequence of the early stage of development of the devices is that the software support package is currently only available for ARM which brings me to the actual topic of this post.
Since AFL has been my go-to fuzzing framework for quite a while but I have only used it for Linux x86 code until now this was the perfect opportunity for me to explore AFL's capabilities for fuzzing ARM binaries.

AFL on the Raspberry Pi

For working with ARM binaries in AFL there are basically two options:

AFL's QEMU Mode, or
running AFL natively on an ARM host

We will start with the latter by setting up AFL on an Raspberry Pi.

Note: This guide was written for Raspberry Pi OS 2020-02-13 but should apply to most distributions based on Debian 10 (Buster).

Build or install ?

With Raspberry Pi OS you have two options; build AFL from source or just install it as a package.

Using the existing packages is pretty straight forward:

# sudo apt-get install afl afl-clang

This should be everything you need to get going.

If on the other hand you decide to build AFL from source, e.g. because you want to use some additional patches there is a small obstacle to overcome.
AFL 2.52b currently only supports Clang 6, the default version for Debian 10 is Clang 7, though. Therefore to build AFL you will first have to install Clang 6:

# sudo apt-get install clang-6.0

Additionally you need to supply some additional flags when compiling AFL:

# cd afl-2.52b
# AFL_NO_X86=1 make
# make -C llvm_mode LLVM_CONFIG=llvm-config-6.0 CC=clang-6.0

That should do the trick.

Compiling with afl-clang-fast and running afl-fuzz

If you are fuzzing an application for which you have the source code you can now recompile it with instrumentation by using afl-clang-fast as a drop in replacement for gcc or clang.
Depending on your build system one of the following is probably all you need:

# make CC=/usr/bin/afl-clang-fast CXX=/usr/bin/afl-clang-fast++
... or ...
# CC=/usr/bin/afl-clang-fast CXX=/usr/bin/afl-clang-fast++ cmake ...

Once your target binary is ready you should be able to run afl-fuzz just as you would on x86, e.g. as:

# afl-fuzz -i /tmp/in -o /tmp/out -- ./foo @@

Fuzzing ARM binaries on x86 with AFL's QEMU mode

The second option when working with ARM binaries is to not use an actual ARM target, but to run afl-fuzz on an x86 host in QEMU mode.

Building AFL with QEMU support

Building AFL with QEMU support should in theory also be straight forward:

# cd afl-2.52b
# make
# cd qemu_mode
# ./build_qemu_support.sh

Unfortunately I ran into several issues with the last build step on Ubuntu 20.04 LTS.

The first error I encountered was this one:

CC      util/memfd.o
util/memfd.c:40:12: error: static declaration of ‘memfd_create’ follows non-static declaration
   40 | static int memfd_create(const char *name, unsigned int flags)
      |            ^~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/mman-linux.h:113,
                 from /usr/include/x86_64-linux-gnu/bits/mman.h:34,
                 from /usr/include/x86_64-linux-gnu/sys/mman.h:41,
                 from /media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/include/sysemu/os-posix.h:29,
                 from /media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/include/qemu/osdep.h:104,
                 from util/memfd.c:28:
/usr/include/x86_64-linux-gnu/bits/mman-shared.h:50:5: note: previous declaration of ‘memfd_create’ was here
   50 | int memfd_create (const char *__name, unsigned int __flags) __THROW;
      |     ^~~~~~~~~~~~
make: *** [/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/rules.mak:66: util/memfd.o] Error 1

You can find my patch to solve this problem in memfd.diff.
The patch file has to be placed in afl-2.52b/qemu_mode/patches/ and appended to the list of patches to be applied by the build script as follows:

--- a/build_qemu_support.sh  2020-08-07 18:44:29.794033735 +0200
+++ b/build_qemu_support.sh       2020-08-07 18:44:42.801908141 +0200
@@ -131,6 +131,7 @@
 patch -p1 <../patches/elfload.diff || exit 1
 patch -p1 <../patches/cpu-exec.diff || exit 1
 patch -p1 <../patches/syscall.diff || exit 1
+patch -p1 <../patches/memfd.diff || exit 1

With this problem out of the way I ran into the next error:

  CC      x86_64-linux-user/linux-user/syscall.o
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/syscall.c:261:16: error: static declaration of ‘gettid’ follows non-static declaration
  261 | _syscall0(int, gettid)
      |                ^~~~~~
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/syscall.c:191:13: note: in definition of macro ‘_syscall0’
  191 | static type name (void)   \
      |             ^~~~
In file included from /usr/include/unistd.h:1170,
                 from /media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/include/qemu/osdep.h:75,
                 from /media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/syscall.c:20:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: previous declaration of ‘gettid’ was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/ioctls.h:173:9: error: ‘SIOCGSTAMP’ undeclared here (not in a function); did you mean ‘SIOCSRARP’?
  173 |   IOCTL(SIOCGSTAMP, IOC_R, MK_PTR(MK_STRUCT(STRUCT_timeval)))
      |         ^~~~~~~~~~
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/syscall.c:5597:23: note: in definition of macro ‘IOCTL’
 5597 |     { TARGET_ ## cmd, cmd, #cmd, access, 0, {  __VA_ARGS__ } },
      |                       ^~~
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/ioctls.h:174:9: error: ‘SIOCGSTAMPNS’ undeclared here (not in a function); did you mean ‘SIOCGSTAMP_OLD’?
  174 |   IOCTL(SIOCGSTAMPNS, IOC_R, MK_PTR(MK_STRUCT(STRUCT_timespec)))
      |         ^~~~~~~~~~~~
/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/linux-user/syscall.c:5597:23: note: in definition of macro ‘IOCTL’
 5597 |     { TARGET_ ## cmd, cmd, #cmd, access, 0, {  __VA_ARGS__ } },
      |                       ^~~
make[1]: *** [/media/ramdisk/afl-2.52b/qemu_mode/qemu-2.10.0/rules.mak:66: linux-user/syscall.o] Error 1
make: *** [Makefile:326: subdir-x86_64-linux-user] Error 2

This problem is solved by syscall.diff which replaces the existing syscall patch in afl-2.52b/qemu_mode/patches/syscall.diff.

With these patches you should now have afl-fuzz with working QEMU mode.

Running AFL in QEMU mode

Unless your target binary was statically linked you will probably have to set a library path in your QEMU environment so that QEMU finds the library dependencies of your binary:

# export QEMU_SET_ENV=LD_LIBRARY_PATH=/path/to/your/arm/libs/

You can then run AFL in QEMU mode with the -Q flag, e.g:

# afl-fuzz -Q -i /tmp/in -o /tmp/out ./foo.arm @@

Doing so I encountered the following message:

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (200 MB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

      ( ulimit -Sv $[199 << 10]; /path/to/fuzzed_app )

      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.

    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke  for troubleshooting tips.

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2253

While the documentation says that this is to be expected since QEMU uses quite a bit of memory, for me the error was not solved by just raising the memory limit.
Even at 4GB I kept getting the same error:

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (3.91 GB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

      ( ulimit -Sv $[3999 << 10]; /path/to/fuzzed_app )

      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.

    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke  for troubleshooting tips.

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2253

The solution here was set the following environment variable:

# export QEMU_RESERVED_VA=0xf00000

Conclusion

Now that you know two ways to use afl-fuzz with ARM binaries you will probably ask: "Which one should I choose?"
And the answer to that clearly is: "It depends."

Personally I will continue using both since they cover different use cases.
As long as I have not reverse engineered the communication protocol of the above mentioned devices far enough to be able to build a stub driver for fuzzing I need to run a test loop that talks to the actual hardware to get sufficient code coverage.
While this is painfully slow it is much easier done on the Raspberry Pi since I have easy access to the I²C bus there.
The tests for all the functions that do not need to talk to hardware on the other hand I run on my x86 Workstation in QEMU mode since that is a lot faster than running the same tests on the Raspberry Pi.

Finally to give you a rough idea of the speed difference, here are to screenshots of alf-fuzz running the same binary, first on the Raspberry Pi:
afl-fuzz running on a Raspberry Pi B
And here on an i3-3320 in QEMU:
afl-fuzz QEMU mode