Paper 2022/1699

SoK: Use of Cryptography in Malware Obfuscation

Hassan Asghar, Macquarie University
Benjamin Zi Hao Zhao, Macquarie University
Muhammad Ikram, Macquarie University
Giang Nguyen, Macquarie University
Dali Kaafar, Macquarie University
Sean Lamont, Defence Science and Technology Group
Daniel Coscia, Defence Science and Technology Group

We look at the use of cryptography to obfuscate malware. Most surveys on malware obfuscation only discuss simple encryption techniques (e.g., XOR encryption), which are easy to defeat (in principle), since the decryption algorithm and the key is shipped within the program. This SoK proposes a principled definition of malware obfuscation, and categorises instances of malware obfuscation that use cryptographic tools into those which evade detection and those which are detectable. The SoK first examines easily detectable schemes such as string encryption, class encryption and XOR encoding, found in most obfuscated malware. It then details schemes that can be shown to be hard to break, such as the use of environmental keying. We also analyse formal cryptographic obfuscation, i.e., the notions of indistinguishability and virtual black box obfuscation, from the lens of our proposed model on malware obfuscation.

Available format(s)
Publication info
malware obfuscation environmental keying
Contact author(s)
hassan asghar @ mq edu au
ben_zi zhao @ mq edu au
muhammad ikram @ mq edu au
duclinhgiang nguyen @ hdr mq edu au
dali kaafar @ mq edu au
sean lamont2 @ dst defence gov au
daniel coscia1 @ dst defence gov au
2022-12-10: approved
2022-12-07: received
See all versions
Short URL
Creative Commons Attribution


      author = {Hassan Asghar and Benjamin Zi Hao Zhao and Muhammad Ikram and Giang Nguyen and Dali Kaafar and Sean Lamont and Daniel Coscia},
      title = {SoK: Use of Cryptography in Malware Obfuscation},
      howpublished = {Cryptology ePrint Archive, Paper 2022/1699},
      year = {2022},
      note = {\url{}},
      url = {}
Note: In order to protect the privacy of readers, does not use cookies or embedded third party content.