Neurosymbolic Repair of Test Flakiness
This program is tentative and subject to change.
Test flakiness, a non-deterministic behavior of builds irrelevant to code changes, is a major and continuing impediment to delivering reliable software. The very few techniques for the automated repair of test flakiness are specifically crafted to repair either Order-Dependent (OD) or Implementation-Dependent (ID) flakiness. They are also all symbolic approaches, i.e., leverage program analysis to detect and repair known test flakiness patterns and root causes, failing to generalize. To bridge the gap, we propose FlakyDoctor, a neuro-symbolic technique that combines the power of LLMs—generalizability—and program analysis—soundness—to fix different types of test flakiness. Our extensive evaluation using 873 confirmed flaky tests (332 OD and 541 ID) from 243 real-world projects demonstrates the ability of FlakyDoctor in repairing flakiness, achieving 57% (OD) and 59% (ID) success rate. Comparing to three alternative flakiness repair approaches, FlakyDoctor can repair 8% more ID tests than DexFix, 12% more OD flaky tests than ODRepair, and 17% more OD flaky tests than iFixFlakies. Regardless of underlying LLM, the non-LLM components of FlakyDoctor contribute to 12–31 % of the overall performance, i.e., while part of the FlakyDoctor power is from using LLMs, they are not good enough to repair flaky tests in real-world projects alone. What makes the proposed technique superior to related research on test flakiness mitigation specifically and program repair, in general, is repairing 79 previously unfixed flaky tests in real-world projects. We opened pull requests for all cases with corresponding patches; 19 of them were accepted and merged at the time of submission.
This program is tentative and subject to change.
Thu 19 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 11:50 | |||
10:30 20mTalk | Automating Zero-Shot Patch Porting for Hard Forks Technical Papers Shengyi Pan Zhejiang University, You Wang Zhejiang University, Zhongxin Liu Zhejiang University, Xing Hu Zhejiang University, Xin Xia Huawei Technologies, Shanping Li Zhejiang University DOI Pre-print | ||
10:50 20mTalk | Benchmarking Automated Program Repair: An Extensive Study on Both Real-World and Artificial Bugs Technical Papers Yicheng Ouyang University of Illinois at Urbana-Champaign, Jun Yang UIUC, Lingming Zhang University of Illinois Urbana-Champaign DOI | ||
11:10 20mTalk | Neurosymbolic Repair of Test Flakiness Technical Papers Yang Chen University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign | ||
11:30 20mTalk | AutoCodeRover: Autonomous Program Improvement Technical Papers Yuntong Zhang National University of Singapore, Haifeng Ruan National University of Singapore, Zhiyu Fan National University of Singapore, Abhik Roychoudhury National University of Singapore |