Revisiting Test-Case Prioritization on Long-Running Test Suites
The prolonged continuous integration (CI) runs are affecting timely feedback to software developers.
Test-case prioritization (TCP) aims to expose faults sooner by reordering tests such that the ones more likely to fail are run earlier.
TCP is thus especially important for long-running test suites.
While many studies have explored TCP, they are based on outdated CI builds from over 10 years ago with test suites that last several minutes, or builds from inaccessible, proprietary projects.
In this paper, we present LRTS, the first dataset of long-running test suites, with 21,255 CI builds and 57,437 test-suite runs from 10 large-scale, open-source projects that use Jenkins CI.
LRTS spans from 2020 to 2023, with an average test-suite run duration of 6.5 hours.
On LRTS, we study the effectiveness of 59 leading TCP techniques, the impact of confounding test failures on TCP, and TCP for failing tests with no prior failures.
We revisit prior key findings (9 confirmed, 2 refuted) and establish 3 new findings.
Our results show that prioritizing faster tests that recently failed performs the best, outperforming the sophisticated techniques.
Fri 20 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
13:30 - 14:50 | |||
13:30 20mTalk | Segment-Based Test Case Prioritization: A Multi-objective Approach Technical Papers Hieu Huynh Katalon, Nhu Pham Katalon; Ho Chi Minh City University of Science, Tien N. Nguyen University of Texas at Dallas, Vu Nguyen Katalon; Ho Chi Minh City University of Science; Vietnam National University DOI | ||
13:50 20mTalk | Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable Systems Technical Papers Chuan Luo Beihang University, Shuangyu Lyu Beihang University, Qiyuan Zhao National University of Singapore, Wei Wu Central South University; Xiangjiang Laboratory, Hongyu Zhang Chongqing University, Chunming Hu Beihang University DOI | ||
14:10 20mTalk | Synthesis-Based Enhancement for GUI Test Case Migration Technical Papers Yakun Zhang Peking University, Qihao Zhu DeepSeek-AI, Jiwei Yan Institute of Software at Chinese Academy of Sciences, Chen Liu Peking University, Wenjie Zhang Peking University, Yifan Zhao Peking University, Dan Hao Peking University, Lu Zhang Peking University DOI | ||
14:30 20mTalk | Revisiting Test-Case Prioritization on Long-Running Test Suites Technical Papers Runxiang Cheng University of Illinois at Urbana-Champaign, Shuai Wang University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign DOI | ||