Paper 2019/947

nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data

Fabian Boemer, Anamaria Costache, Rosario Cammarota, and Casimir Wierzynski


In previous work, Boemer et al. introduced nGraph-HE, an extension to the Intel nGraph deep learning (DL) compiler, that en- ables data scientists to deploy models with popular frameworks such as TensorFlow and PyTorch with minimal code changes. However, the class of supported models was limited to relatively shallow networks with polynomial activations. Here, we introduce nGraph-HE2, which extends nGraph-HE to enable privacy-preserving inference on standard, pre-trained models using their native activation functions and number fields (typically real numbers). The proposed framework leverages the CKKS scheme, whose support for real numbers is friendly to data science, and a client-aided model to compute activation functions. We first present CKKS-specific optimizations, enabling a 3x-88x runtime speedup for scalar encoding, and doubling the throughput through a novel use of CKKS plaintext packing into complex numbers. Second, we optimize ciphertext-plaintext addition and multiplication, yielding 2.6x- 4.2x runtime speedup. Third, we present two graph-level optimizations: lazy rescaling and depth-aware encoding. Together, these optimizations enable state-of-the-art throughput of 1,998 images/s on the CryptoNets network. We also present homomorphic evaluation of (to our knowledge) the largest network to date, namely, pre-trained MobileNetV2 models on the ImageNet dataset, with 60.4%/82.7% top-1/top-5 accuracy and an amortized runtime of 381 ms/image.

Available format(s)
Publication info
Published elsewhere.
Privacy-Preserving Machine LearningDeep LearningGraph CompilersHomomorphic Encryption
Contact author(s)
fabian boemer @ intel com
anamaria costache @ intel com
2019-08-29: revised
2019-08-20: received
See all versions
Short URL
Creative Commons Attribution


      author = {Fabian Boemer and Anamaria Costache and Rosario Cammarota and Casimir Wierzynski},
      title = {nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data},
      howpublished = {Cryptology ePrint Archive, Paper 2019/947},
      year = {2019},
      note = {\url{}},
      url = {}
Note: In order to protect the privacy of readers, does not use cookies or embedded third party content.