Revisiting Test-Case Prioritization on Long-Running Test Suites
This program is tentative and subject to change.
The prolonged continuous integration (CI) runs are affecting timely feedback to software developers. Test-case prioritization (TCP) aims to expose bugs sooner by reordering tests in the test suite such that the ones more likely to fail are run earlier. TCP is thus especially important for long-running test suites. While many studies have explored TCP, they use datasets of outdated CI builds (that are mostly over 10 years old) with short-running test suites (that last several minutes on average) or of outdated builds from inaccessible, proprietary projects. In this paper, we present LRTS, the first dataset of long-running test suites, with 21,255 CI builds and 57,437 test-suite runs from 10 large-scale, open-source projects that use Jenkins CI. LRTS spans from 2020 to 2023, with an average test-suite run duration of 6.5 hours. With LRTS, we extensively study 59 leading TCP techniques to identify the best ones on: longer-running test suites, TCP under the impact of confounding test failures, and cases where most tests have no prior failures. Our analyses revisit prior key findings (9 confirmed, 2 refuted) and establish 3 new findings.
This program is tentative and subject to change.
Fri 20 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
13:30 - 14:50 | |||
13:30 20mTalk | Segment-based Test Case Prioritization: a Multi-objective Approach Technical Papers Hieu Huynh Katalon LLC., Nhu Pham University of Science, VNU-HCM; Katalon LLC., Vu Nguyen University of Science, VNU-HCM; Katalon LLC., Tien N. Nguyen University of Texas at Dallas | ||
13:50 20mTalk | Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable Systems Technical Papers Chuan Luo Beihang University, Shuangyu Lyu Beihang University, Qiyuan Zhao National University of Singapore, Wei Wu L3S Research Center, Leibniz University Hannover, Germany, Hongyu Zhang Chongqing University, Chunming Hu Beihang University | ||
14:10 20mTalk | Synthesis-based Enhancement for GUI Test Case Migration Technical Papers Yakun Zhang Peking University, Qihao Zhu Peking University, Jiwei Yan Institute of Software at Chinese Academy of Sciences, China, Chen Liu Clarkson University, Wenjie Zhang Peking University, Yifan Zhao Peking University, Dan Hao Peking University, Lu Zhang Peking University | ||
14:30 20mTalk | Revisiting Test-Case Prioritization on Long-Running Test Suites Technical Papers Runxiang Cheng University of Illinois at Urbana-Champaign, Shuai Wang University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign |