Paper 2026/932

Zephyr: GPU-Efficient Homomorphic Encryption for Privacy-Preserving Transformer Inference

Abstract

Privacy-preserving machine learning (PPML) enables inference over sensitive data without exposing raw inputs, with CKKS being a widely adopted scheme for approximate arithmetic. However, existing CKKS implementations are primarily based on 64-bit residue number system (RNS) representations, creating a mismatch with modern GPUs optimized for 32-bit integer arithmetic. This mismatch introduces substantial computational overhead, limiting the practicality of encrypted transformer inference. In this work, we present Zephyr, a GPU-efficient framework for homomorphic transformer inference via 32-bit arithmetic and grafting. Zephyr revisits the design of CKKS under GPU constraints and introduces a grafting-based representation that decouples scale management from the modulus chain. By constructing the RNS basis entirely with 30-bit primes and managing scale through auxiliary graft structures, Zephyr enables flexible rescaling while remaining compatible with efficient 32-bit GPU execution. Compared to Cheddar (Choi et al., ASPLOS’26), a representative GPU-oriented CKKS design based on fixed 25-30 prime systems, our approach simplifies modulus management and enables more flexible operations across different levels, while reducing rescaling overhead at the cost of additional convolution overhead. We further optimize ciphertext-ciphertext matrix multiplication (CCMM), a major bottleneck in encrypted transformer inference, by eliminating redundant linear transformations and merging overlapping rotation patterns in attention computation. Our theoretical and empirical analysis demonstrates that grafting-based 32-bit CKKS provides a practical and flexible design point for GPU-accelerated PPML inference.