Automated Deep Learning Optimization via DSL-Based Source Code Transformation
As deep learning models become increasingly bigger and more complex, it is critical to improve model training and inference efficiency. Though a variety of highly optimized libraries and packages (known as DL kernels) have been developed, it is tedious and time-consuming to figure out which kernel to use, where to use, and how to use them correctly. To address this challenge, we propose an Automated Deep learning OPTimization approach called Adopter. We design a Domain-Specific Language (DSL) to represent DL model architectures and leverage this DSL to specify model transformation rules required to integrate a DL kernel into a model. Given the source code of a DL model and the transformation rules for a set of kernels, Adopter first performs inter-procedural analysis to identify and express the model architecture in our DSL. Then, Adopter performs scope analysis and sub-sequence matching to identify locations in the model architecture where the transformation rules can be applied. Finally, Adopter proposes a synthesis-based code transformation method to apply the transformation rule. We curated a benchmark with 199 models from Hugging Face and a diverse set of DL kernels. We found that, compared to a state-of-the-art automated code transformation technique, Adopter helps improve the precision and recall by 3% and 56%, respectively. An in-depth analysis of 9 models revealed that on average, Adopter improved the training speed by 22.7% while decreasing the GPU memory usage by 10.5%.
Thu 19 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
15:30 - 17:10 | |||
15:30 20mTalk | One-to-One or One-to-Many? Suggesting Extract Class Refactoring Opportunities with Intra-class Dependency Hypergraph Neural Network Technical Papers Di Cui Xidian University, Qiangqiang Wang Xidian University, Yutong Zhao University of Central Missouri, Jiaqi Wang Xidian University, Minjie Wei Xidian University, Jingzhao Hu Xidian University, Luqiao Wang Xidian University, Qingshan Li Xidian University DOI | ||
15:50 20mTalk | CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature Technical Papers Chenyan Liu Shanghai Jiao Tong University; National University of Singapore, Yufan Cai Shanghai Jiao Tong University; National University of Singapore, Yun Lin Shanghai Jiao Tong University, Yuhuan Huang Shanghai Jiao Tong University, Yunrui Pei Shanghai Jiao Tong University, Bo Jiang Bytedance Network Technology, Ping Yang Bytedance Network Technology, Jin Song Dong National University of Singapore, Hong Mei Shanghai Jiao Tong University DOI | ||
16:10 20mTalk | Arfa: An Agile Regime-Based Floating-Point Optimization Approach for Rounding Errors Technical Papers Jinchen Xu Information Engineering University, Mengqi Cui Information Engineering University, Fei Li Information Engineering University, Zuoyan Zhang Hunan University, Hongru Yang Information Engineering University, Bei Zhou Information Engineering University, Jie Zhao Hunan University DOI | ||
16:30 20mTalk | Automated Deep Learning Optimization via DSL-Based Source Code Transformation Technical Papers Ruixin Wang Purdue University, Minghai Lu Purdue University, Cody Hao Yu BosonAI, Yi-Hsiang Lai Amazon Web Services, Tianyi Zhang Purdue University DOI | ||