Paper 2026/503

SwiftSNNI: Optimized Scheduling for Secure Neural Network Inference (SNNI) on Multi-Core Systems

Abstract

Secure Neural Network Inference (SNNI) enables privacy-preserving inference on encrypted data with strong cryptographic guarantees. However, practical deployments suffer from high preprocessing overhead, significant communication costs, and sequential execu- tion. These limitations lead to low throughput, underutilized system resources, long queueing delays, and poor scalability. This work introduces SwiftSNNI, a unified, resource-aware scheduling framework for SNNI. It implements a hybrid offline–online strategy that orchestrates offline preprocessing (𝑇pre,𝑖 ) and online inference (𝑇on,𝑖 ) jobs to maximize parallelism. By formulating SNNI scheduling as a constrained optimization problem, SwiftSNNI overlaps 𝑇pre, i job execution of future requests with active 𝑇on, j jobs. SwiftSNNI also incorporates optional advance notices to enable proactive 𝑇pre,𝑖 , which further reduces average input delay (𝐷). Evaluations using five benchmark neural networks (M1, M2, HiNet, AlexNet, VGG-16) under diverse workloads and stochastic arrival rates confirm substantial performance gains. Compared to a parallelized sequential baseline (MS-SHARK), SwiftSNNI achieves up to 97% lower average input delay (𝐷), up to an 81% reduction in makespan (≈ 5.4× speed up), and delivers up to a 5.6× increase in throughput. Furthermore, SwiftSNNI reduces average waiting time (𝑊 ) by up to 99.7%, demonstrating robust starvation prevention for high concurrency workloads. SwiftSNNI supports concurrent execution, scales to larger neural networks, and provides an efficient runtime for SNNI practical deployments. SwiftSNNI’s source code is available online at: https://github.com/KanwalBat00l/SwiftSNNI