Paper 2017/138

How (not) to Use Welch's T-test in Side-Channel Security Evaluations

François-Xavier Standaert


The Test Vector Leakage Assessment (TVLA) methodology is a qualitative tool relying on Welch's T-test to assess the security of cryptographic implementations against side-channel attacks. Despite known limitations (e.g., risks of false negatives and positives), it is sometimes considered as a pass-fail test to determine whether such implementations are "safe" or not (without clear definition of what is "safe"). In this note, we clarify the limited quantitative meaning of this test when used as a standalone tool. For this purpose, we first show that the straightforward application of this approach to assess the security of a masked implementation is not sufficient. More precisely, we show that even in a simple (more precisely, univariate) case study that seems best suited for the TVLA methodology, detection (or lack thereof) with Welch's T-test can be totally disconnected from the actual security level of an implementation. For this purpose, we put forward the case of a realistic masking scheme that looks very safe from the TVLA point-of-view and is nevertheless easy to break. We then discuss this result in more general terms and argue that this limitation is shared by all "moment-based" security evaluations. We conclude the note positively, by describing how to use moment-based analyzes as a useful ingredient of side-channel security evaluations, to determine a "security order".

Available format(s)
Publication info
Published elsewhere. Proceedings of CARDIS 2018
side-channel analysissecurity evaluations
Contact author(s)
fstandae @ uclouvain be
2018-10-15: revised
2017-02-20: received
See all versions
Short URL
Creative Commons Attribution


      author = {François-Xavier Standaert},
      title = {How (not) to Use Welch's T-test in Side-Channel Security Evaluations},
      howpublished = {Cryptology ePrint Archive, Paper 2017/138},
      year = {2017},
      note = {\url{}},
      url = {}
Note: In order to protect the privacy of readers, does not use cookies or embedded third party content.