Fuzzing is an efficient method for locating software program vulnerabilities. Over the previous few years Android has been targeted on bettering the effectiveness, scope, and comfort of fuzzing throughout the group. This effort has straight resulted in improved check protection, fewer safety/stability bugs, and better code high quality. Our implementation of steady fuzzing permits software program groups to seek out new bugs/vulnerabilities, and forestall regressions robotically with out having to manually provoke fuzzing runs themselves. This submit recounts a short historical past of fuzzing on Android, shares how Google performs fuzzing at scale, and paperwork our expertise, challenges, and success in constructing an infrastructure for automating fuzzing throughout Android. When you’re serious about contributing to fuzzing on Android, we’ve included directions on the best way to get began, and data on how Android’s VRP rewards fuzzing contributions that discover vulnerabilities.

A Temporary Historical past of Android Fuzzing

Fuzzing has been round for a few years, and Android was among the many early giant software program tasks to automate fuzzing and prioritize it equally to unit testing as a part of the broader aim to make Android essentially the most safe and secure working system. In 2019 Android kicked off the fuzzing mission, with the aim to assist institutionalize fuzzing by making it seamless and a part of code submission. The Android fuzzing mission resulted in an infrastructure consisting of Pixel telephones and Google cloud based mostly digital units that enabled scalable fuzzing capabilities throughout the whole Android ecosystem. This mission has since grown to turn out to be the official inner fuzzing infrastructure for Android and performs 1000’s of fuzzing hours per day throughout a whole lot of fuzzers.

Beneath the Hood: How Is Android Fuzzed

Step 1: Outline and discover all of the fuzzers in Android repo

Step one is to combine fuzzing into the Android construct system (Soong) to allow construct fuzzer binaries. Whereas builders are busy including options to their codebase, they will embody a fuzzer to fuzz their code and submit the fuzzer alongside the code they’ve developed. Android Fuzzing makes use of a construct rule referred to as cc_fuzz (see instance beneath). cc_fuzz (we additionally help rust_fuzz and java_fuzz) defines a Soong module with supply file(s) and dependencies that may be constructed right into a binary.

cc_fuzz {
  identify: "fuzzer_foo",

  srcs: [
    "fuzzer_foo.cpp",
  ],

  static_libs: [
    "libfoo",
  ],

  host_supported: true,
}

A packaging rule in Soong finds all of those cc_fuzz definitions and builds them robotically. The precise fuzzer construction itself could be very easy and consists of 1 fundamental methodology (LLVMTestOneInput):

#embody <stddef.h>
#embody <stdint.h>

extern "C" int LLVMFuzzerTestOneInput(
               const uint8_t *knowledge,
               size_t measurement) {

  // Right here you invoke the code to be fuzzed. 
  return 0;
}

This fuzzer will get robotically constructed right into a binary and together with its static/dynamic dependencies (as specified within the Android construct file) are packaged into a zipper file which will get added to the principle zip containing all fuzzers as proven within the instance beneath.

Step 2: Ingest all fuzzers into Android builds

As soon as the fuzzers are discovered within the Android repository and they’re constructed into binaries, the subsequent step is to add them to the cloud storage in preparation to run them on our backend. This course of is run a number of occasions each day. The Android fuzzing infrastructure makes use of an open supply steady fuzzing framework (Clusterfuzz) to run fuzzers repeatedly on Android units and emulators. To be able to run the fuzzers on clusterfuzz, the fuzzers zip recordsdata are renamed after the construct and the newest construct will get to run (see diagram beneath):

The fuzzer zip file accommodates the fuzzer binary, corresponding dictionary in addition to a subfolder containing its dependencies and the git revision numbers (sourcemap) equivalent to the construct. Sourcemaps are used to boost stack traces and produce crash studies.

Step 3: Run fuzzers repeatedly and discover bugs

Operating fuzzers repeatedly is finished by means of scheduled jobs the place every job is related to a set of bodily units or emulators. A job can be backed by a queue that represents the fuzzing duties that should be run. These duties are a mix of operating a fuzzer, reproducing a crash present in an earlier fuzzing run, or minimizing the corpus, amongst different duties.

Every fuzzer is run for a number of hours, or till they discover a crash. After the run, Android fuzzing takes the entire fascinating enter found in the course of the run and provides it to the fuzzer corpus. This corpus is then shared throughout fuzzer runs and grows over time. The fuzzer is then prioritized in subsequent runs in line with the expansion of latest protection and crashes discovered (if any). This ensures we offer the best fuzzers extra time to run and discover fascinating crashes.

Step 4: Generate fuzzers line protection

What good is a fuzzer if it’s not fuzzing the code you care about? To enhance the standard of the fuzzer and to observe the general progress of Android fuzzing, two kinds of protection metrics are calculated and out there to Android builders. The primary metric is for edge protection which refers to edges within the Management Circulation Graph (CFG). By instrumenting the fuzzer and the code being fuzzed, the fuzzing engine can observe small snippets of code that get triggered each time execution stream reaches them. That approach, fuzzing engines know precisely what number of (and what number of occasions) every of those instrumentation factors bought hit on each run to allow them to combination them and calculate the protection.

INFO: Seed: 2859304549
INFO: Loaded 1 modules   (773 inline 8-bit counters): 773 [0x5610921000, 0x5610921305),
INFO: Loaded 1 PC tables (773 PCs): 773 [0x5610921308,0x5610924358),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 2 ft: 2 corp: 1/1b lim: 4 exec/s: 0 rss: 24Mb
#413    NEW    cov: 3 ft: 3 corp: 2/9b lim: 8 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 InsertRepeatedBytes-
#3829   NEW    cov: 4 ft: 4 corp: 3/17b lim: 38 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 ChangeBinInt-
...

Line coverage inserts instrumentation points specifying lines in the source code. Line coverage is very useful for developers as they can pinpoint areas in the code that are not covered and update their fuzzers accordingly to hit those areas in future fuzzing runs.

Drilling into any of the folders can show the stats per file:

Further clicking on one of the files shows the lines that were touched and lines that never got coverage. In the example below, the first line has been fuzzed ~5 million times, but the fuzzer never makes it into lines 3 and 4, indicating a gap in the coverage for this fuzzer.

We have dashboards internally that measure our fuzzing coverage across our entire codebase. In order to generate these coverage dashboards yourself, you follow these steps.

Another measurement of the quality of the fuzzers is how many fuzzing iterations can be done in one second. It has a direct relationship with the computation power and the complexity of the fuzz target. However, this parameter alone can not measure how good or effective the fuzzing is.

How we handle fuzzer bugs

Android fuzzing utilizes the Clusterfuzz fuzzing infrastructure to handle any found crashes and file a ticket to the Android security team. Android security makes an assessment of the crash based on the Android Severity Guidelines and then routes the vulnerability to the proper team for remediation. This entire process of finding the reproducible crash, routing to Android Security, and then assigning the issue to a team responsible can take as little as two hours, and up to a week depending on the type of crash and the severity of the vulnerability.

One example of a recent fuzzer success is (CVE 2022-20473), where an internal team wrote a 20-line fuzzer and submitted it to run on Android fuzzing infra. Within a day, the fuzzer was ingested and pushed to our fuzzing infrastructure to begin fuzzing, and shortly found a critical severity vulnerability! A patch for this CVE has been applied by the service team.

Why Android Continues to Invest in Fuzzing

Protection Against Code Regressions

The Android Open Source Project (AOSP) is a large and complex project with many contributors. As a result, there are thousands of changes made to the project every day. These changes can be anything from small bug fixes to large feature additions, and fuzzing helps to find vulnerabilities that may be inadvertently introduced and not caught during code review.

Continuous fuzzing has helped to find these vulnerabilities before they are introduced in production and exploited by attackers. One real-life example is (CVE-2023-21041), a vulnerability discovered by a fuzzer written three years ago. This vulnerability affected Android firmware and could have led to local escalation of privilege with no additional execution privileges needed. This fuzzer was running for many years with limited findings until a code regression led to the introduction of this vulnerability. This CVE has since been patched.

Protection against unsafe memory language pitfalls

Android has been a huge proponent of Rust, with Android 13 being the first Android release with the majority of new code in a memory safe language. The amount of new memory-unsafe code entering Android has decreased, but there are still millions of lines of code that remain, hence the need for fuzzing persists.

No One Code is Safe: Fuzzing code in memory-safe languages

Our work does not stop with non-memory unsafe languages, and we encourage fuzzer development in languages like Rust as well. While fuzzing won’t find common vulnerabilities that you would expect to see memory unsafe languages like C/C++, there have been numerous non-security issues discovered and remediated which contribute to the overall stability of Android.

Fuzzing Challenges

In addition to generic C/C++ binaries issues such as missing dependencies, fuzzers can have their own classes of problems:

Low executions per second: in order to fuzz efficiently, the number of mutations has to be in the order of hundreds per second otherwise the fuzzing will take a very long time to cover the code. We addressed this issue by adding a set of alerts that continuously monitor the health of the fuzzers as well as any sudden drop in coverage. Once a fuzzer is identified as underperforming, an automated email is sent to the fuzzer author with details to help them improve the fuzzer.

Fuzzing the wrong code: Like all resources, fuzzing resources are limited. We want to ensure that those resources give us the highest return, and that generally means devoting them towards fuzzing code that processes untrusted (i.e. potentially attacker controlled) inputs. This can cover any way that the phone can receive input including Bluetooth, NFC, USB, web, etc. Parsing structured input is particularly interesting since there is room for programming errors due to specs complexity. Code that generates output is not particularly interesting to fuzz. Similarly internal code that is not exposed publicly is also less of a security concern. We addressed this issue by identifying the most vulnerable code (see the following section).

What to fuzz

In order to fuzz the most important components of the Android source code, we focus on libraries that have:

1. A history of vulnerabilities: the history should not be the distant history since context change but more focus on the last 12 months.
2. Recent code changes: research indicates that more vulnerabilities are found in recently changed code than code that is more stable.
3. Remote access: vulnerabilities in code that are reachable remotely can be critical.
4. Privileged: Similarly to #3, vulnerabilities in code that runs in privileged processes can be critical.

How to submit a fuzzer to AOSP

We’re constantly writing and improving fuzzers internally to cover some of the most sensitive areas of Android, but there is always room for improvement. If you’d like to get started writing your own fuzzer for an area of AOSP, you’re welcome to do so to make Android more secure (example CL):

1. Get Android source code
2. Have a testing phone?
3. Write a fuzz target (follow guidelines in ‘What to fuzz’ section)
4. Upload your fuzzer to AOSP.

Get started by reading our documentation on Fuzzing with libFuzzer and check your fuzzer into the Android Open Source project. If your fuzzer finds a bug, you can submit it to the Android Bug Bounty Program and could be eligible for a reward!