Paper 2026/1333

Apples, Oranges, and Signatures: Pitfalls and Methodology in ML-DSA Benchmarking

Abstract

Cryptographic migration, particularly in the post-quantum setting, poses significant practical challenges and requires reliable performance data to support sound engineering decisions. For ML-DSA, however, existing benchmarking practices often produce misleading or non-comparable results, complicating migration and cryptographic agility efforts. This paper analyzes common pitfalls in benchmarking ML-DSA signature operations, including subtle inconsistencies when comparing security levels. We show that execution-time variability of the ML-DSA signing algorithm - an inherent property due to rejection sampling and other data-dependent components - makes commonly used straightforward metrics, e.g., min/average/max, unsuitable for migration planning. To address this gap, we propose a robust benchmarking methodology based on standardized input data sets and clearly qualified reporting metrics. The proposed approach enables fair comparison across hardware and software implementations and supports designers of real-time systems to assess the worst-case execution time.