An Empirical Examination of Fuzzer Mutator Performance
This program is tentative and subject to change.
Over the past decade, hundreds of fuzzers have been published in top-tier security and software engineering conferences. Fuzzers are used to automatically test programs, ideally creating high-coverage input corpora and finding bugs. Modern "greybox'' fuzzers evolve a corpus of inputs by applying \emph{mutations} to inputs and then executing those new inputs while collecting coverage. New inputs that are "interesting'' (e.g. reveal new coverage) are saved to the corpus. Given their non-deterministic nature, the impact of each design decision on the fuzzer’s performance can be difficult to predict. Some design decisions (e.g. "should the fuzzer perform deterministic mutations of inputs'') are exposed to end-users as configuration flags, but others (e.g. "what kinds of random mutations to apply to inputs?'') are typically baked-in to the fuzzer code itself. This paper describes our over 12.5-CPU-year evaluation of the set of mutation operators employed by the popular AFL++ fuzzer including the \textit{havoc} phase, splicing and \redqueen, exploring the impact of adjusting some of those unexposed configurations.
In this experience paper, we propose a methodology for determining different fuzzers’ behavioral diversity with respect to branch coverage and bug detection using rigorous statistical methods. Our key finding is that, across a range of targets, disabling certain mutation operators (some of which were previously "baked-in'' to the fuzzer) resulted in inputs that cover different lines of code and reveal different bugs. A surprising result is disabling certain mutators leads to \textbf{more diverse} coverage and allows the fuzzer to find \textbf{more} bugs \textbf{faster}. We call for researchers to investigate seemingly simple design decisions in fuzzers more thoroughly and encourage fuzzer developers to expose more configuration parameters pertaining to these design decisions to end-users.
This program is tentative and subject to change.
Wed 18 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 11:50 | |||
10:30 20mTalk | Large Language Models for Equivalent Mutant Detection: How Far are We? Technical Papers Zhao Tian Tianjin University, Honglin Shu Kyushu University, Dong Wang Tianjin University, China, Xuejie Cao College of Intelligence and Computing, Tianjin University, Yasutaka Kamei Kyushu University, Junjie Chen Tianjin University DOI Pre-print | ||
10:50 20mTalk | An Empirical Examination of Fuzzer Mutator Performance Technical Papers James Kukucka George Mason University, Luís Pina University of Illinois at Chicago, Paul Ammann George Mason University, USA, Jonathan Bell Northeastern University | ||
11:10 20mTalk | Equivalent Mutants in the Wild: Identifying and Efficiently Suppressing Equivalent Mutants for Java Programs Technical Papers Benjamin Kushigian University of Washington, Samuel Kaufman University of Washington, Ryan Featherman University of Washington, Hannah Potter University of Washington, Ardi Madadi University of Washington, René Just University of Washington | ||
11:30 20mTalk | LPR: Large Language Models-Aided Program Reduction Technical Papers Mengxiao Zhang University of Waterloo, Yongqiang Tian The Hong Kong University of Science and Technology, Zhenyang Xu University of Waterloo, Yiwen Dong University of Waterloo, Shin Hwei Tan Concordia University, Chengnian Sun University of Waterloo DOI |