Test Selection for Deep Neural Networks using Meta-Models with Uncertainty Metrics
With the use of Deep Learning (DL) in safety-critical domains, the systematic testing of these systems has become a critical issue for human life. Due to the data-driven nature of Deep Neural Networks (DNNs), the effectiveness of tests is closely related to the adequacy of test datasets. Test data need to be labeled, which requires manual human effort and sometimes expert knowledge. DL system testers aim to select the test data that will be most helpful in identifying the weaknesses of the DNN model by using resources efficiently.
To help achieve this goal, we propose a test data prioritization approach based on using a meta-model that gets uncertainty metrics as input, which are derived from outputs of other base models. Integrating different uncertainty metrics helps overcome individual limitations of these metrics and be effective in a wider range of scenarios. We train the meta-models with the objective of predicting whether a test input will lead the tested model to make an incorrect prediction or not. We conducted an experimental evaluation with popular image classification datasets and DNN models to evaluate the proposed approach. The results of the experiments demonstrate that our approach effectively prioritizes the test datasets and outperforms existing state-of-the-art test prioritization methods used in comparison. In the experiments, we evaluated the test prioritization approach from a distribution-aware perspective by generating test datasets with and without out-of-distribution data.
Thu 19 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 11:50 | |||
10:30 20mTalk | Distance-Aware Test Input Selection for Deep Neural Networks Technical Papers Zhong Li , Zhengfeng Xu Nanjing University, Ruihua Ji Nanjing University, Minxue Pan Nanjing University, Tian Zhang Nanjing University, Linzhang Wang Nanjing University, Xuandong Li Nanjing University DOI | ||
10:50 20mTalk | Test Selection for Deep Neural Networks using Meta-Models with Uncertainty Metrics Technical Papers Demet Demir Department of Information Systems, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye, Aysu Betin Can Department of Information Systems, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye, Elif Surer Department of Modeling and Simulation, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye | ||
11:10 20mTalk | Datactive: Data Fault Localization for Object Detection Systems Technical Papers Yining Yin Nanjing University, China, Yang Feng Nanjing University, Shihao Weng Nanjing University, Yuan Yao Nanjing University, Jia Liu Nanjing University, Zhihong Zhao | ||
11:30 20mTalk | TeDA: A Testing Framework for Data Usage Auditing in Deep Learning Model Development Technical Papers Xiangshan Gao Zhejiang University and Huawei Technology, Jialuo Chen Zhejiang University, Jingyi Wang Zhejiang University, Jie Shi Huawei International, Peng Cheng Zhejiang University, Jiming Chen Zhejiang University |