Policy Testing with MDPFuzz (Replicability Study)
This program is tentative and subject to change.
In recent years, following tremendous achievements in Reinforcement Learning, a great deal of interest has been devoted to ML models for sequential decision-making. Together with these scientific breakthroughs/advances, research has been conducted to develop automated functional testing methods for finding faults in black-box Markov decision processes. In 2022, Pang et al. presented a black-box fuzz testing framework called MDPFuzz. The method consists of a fuzzer whose main feature is to use Gaussian Mixture Models (GMMs) to compute coverage of the test inputs as the likelihood to have already observed their results. This guidance through coverage evaluation aims at favoring novelty during testing and fault discovery in the decision model. Pang et al. evaluated their work with four use cases, by comparing the number of failures found after twelve-hour testing campaigns with or without the guidance of the GMMs (ablation study). In this paper, we verify some of the key findings of the original paper and explore the limits of MDPFuzz through reproduction and replication. We re-implemented the proposed methodology and evaluated our replication in a large-scale study that extends the original four use cases with three new ones. Furthermore, we compare MDPFuzz and its ablated counterpart with a random testing baseline. We also assess the effectiveness of coverage guidance for different parameters, something that has not been done in the original evaluation. Despite this parameter analysis and unlike Pang et al .’ original conclusions, we find that in most cases, the aforementioned ablated Fuzzer outperforms MDPFuzz, and conclude that the coverage model proposed does not lead to finding more faults.
This program is tentative and subject to change.
Wed 18 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 11:50 | |||
10:30 20mTalk | Policy Testing with MDPFuzz (Replicability Study) Technical Papers Quentin Mazouni Simula Research Laboratory, Helge Spieker Simula Research Laboratory, Norway, Arnaud Gotlieb Simula Research Laboratory, Mathieu Acher University of Rennes, France / Inria, France / CNRS, France / IRISA, France | ||
10:50 20mTalk | Fuzzing JavaScript Interpreters with Coverage-Guided Reinforcement Learning for LLM-based Mutation Technical Papers | ||
11:10 20mTalk | Enhancing ROS System Fuzzing through Callback Tracing Technical Papers Yuheng Shen Tsinghua University, Jianzhong Liu ShanghaiTech University, Yiru Xu Tsinghua University, Hao Sun ETH Zurich, Mingzhe Wang Tsinghua University, Heyuan Shi Central South University, Nan Guan City University of Hong Kong, Yu Jiang Tsinghua University DOI | ||
11:30 20mTalk | Sleuth: A Switchable Dual-Mode Fuzzer to Investigate Bug Impacts Following a Single PoC Technical Papers Haolai Wei Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, Liwei Chen Institute of Information Engineering, Chinese Academy of Sciences, Zhijie Zhang Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, Gang Shi Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, Dan Meng Institute of Information Engineering, Chinese Academy of Sciences |