Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach
- Resource Type
- Conference
- Authors
- Etter, Brian; Hu, James Lee; Ebrahimi, Mohammadreza; Li, Weifeng; Li, Xin; Chen, Hsinchun
- Source
- 2023 IEEE International Conference on Data Mining Workshops (ICDMW) ICDMW Data Mining Workshops (ICDMW), 2023 IEEE International Conference on. :1313-1321 Dec, 2023
- Subject
- Computing and Processing
Deep learning
Learning systems
Perturbation methods
Reinforcement learning
Detectors
Malware
Engines
Adversarial Robustness
Reinforcement Learning
Adversarial Malware Variants
Adversarial Malware Generation
Obfuscation
- Language
- ISSN
- 2375-9259
Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.