Paper 2026/730

Towards Zero Rotation and Beyond: Architecting Neural Networks for Fast Secure Inference with Homomorphic Encryption

Abstract

Privacy-preserving deep learning addresses privacy concerns in Machine Learning as a Service (MLaaS) using Homomorphic Encryption (HE) for linear computations. Nevertheless, the high computational cost remains a challenge. While prior work has attempted to improve the efficiency, most are built upon models originally designed for plaintext inference. These models are inherently limited by architectural inefficiencies when adapted to HE settings. We argue that substantial efficiency improvements can be achieved by designing networks specifically tailored to the unique computational characteristics of HE, rather than retrofitting existing plaintext models. Our design comprises two main components: the building block and the overall architecture. The first, StriaBlock, targets the most expensive HE operation—Rotation. It integrates ExRot-Free Convolution and a novel Cross Kernel, completely eliminating the need for external Rotation and requiring only 19% of the internal Rotation operations compared to plaintext models. The second component, the architectural principle, includes the Focused Constraint Principle, which limits cost-sensitive factors while preserving flexibility in others, and the Channel Packing-Aware Scaling Principle, which dynamically adapts bottleneck ratios based on ciphertext channel capacity that varies with network depth. These strategies efficiently control the local and overall HE cost, enabling a balanced architecture for HE settings. The resulting network, StriaNet, is comprehensively evaluated. While prior works primarily focus on small-scale datasets such as CIFAR-10, we conduct an extensive evaluation of StriaNet across datasets of varying scales, including large-scale (ImageNet), medium-scale (Tiny ImageNet), and small-scale (CIFAR-10) benchmarks. At comparable accuracy levels, StriaNet achieves speedups of 9.78 times, 6.01 times, and 9.24 times on ImageNet, Tiny ImageNet, and CIFAR-10, respectively.