Paper 2026/1866

Peeling Nonlinear Layers: Algebraic Cryptanalysis of Full-Round Iasta

Abstract

Iasta is a stream cipher designed for hybrid homomorphic encryption (HHE), with claimed $128$-bit security for its Iasta-3 and Iasta-4 instances. In this work, we present the first third-party cryptanalysis of the full-round Iasta-3 and Iasta-4 instances and further extend our approach to Iasta-5. Our cryptanalysis exploits the restricted randomness and structured construction of the nonce-dependent affine-layer matrices. We show that the matrix space contains only $2^{31.30}$ and $2^{19.62}$ distinct matrices for Iasta-3 and Iasta-4/5, respectively, compared with the $2^{29}$ and $2^{22}$ matrix randomness claimed by the designers. This restricted matrix space enables us to construct weak nonces that induce an identical matrix in the final affine layer. For such nonces, we peel off the final Cube transformation, yielding polynomial equations of degree at most $2^{d-1}$ in the secret-key coefficients, which we solve using linearization. We further consider the more restrictive class of nonces that induce identical matrices in both the initial and final affine layers. This allows us to additionally peel off the first non-linear layer, reducing the degree of the resulting equations to at most $2^{d-2}$ at the cost of introducing additional linearization variables. This extended attack substantially improves the attack complexity for Iasta-4 and Iasta-5. For Iasta-3 and Iasta-4, our best estimated attack complexities are $2^{59}$ and $2^{67}$ operations, respectively, under $\omega=2$, reducing the claimed $128$-bit security level to an almost square-root security level. Even under the conservative setting $\omega=3$, the attack requires approximately $2^{80}$ and $2^{82}$ operations against Iasta-3 and Iasta-4, respectively, both below $2^{128}$. For Iasta-5, the extended attack achieves an estimated complexity of $2^{99}$ operations under $\omega=2$. Although no overall security level is explicitly specified for Iasta-5, this result demonstrates that our attack can also reach a complexity below $2^{128}$ for this instance. In all three instances, our attacks reveal a structural weakness in Iasta arising from the restricted space of nonce-dependent affine-layer matrices.