### HyperLogLog: Exponentially Bad in Adversarial Settings

Kenneth G. Paterson and Mathilde Raynal

##### Abstract

Computing the count of distinct elements in large data sets is a common task but naive approaches are memory-expensive. The HyperLogLog (HLL) algorithm (Flajolet et al., 2007) estimates a data set’s cardinality while using significantly less memory than a naive approach, at the cost of some accuracy. This trade-off makes the HLL algorithm very attractive for a wide range of applications such as database management and network monitoring, where an exact count may not be needed. The HLL algorithm and variants of it are implemented in systems such as Redis and Google Big Query. Recently, the HLL algorithm has started to be proposed for use in scenarios where the inputs may be adversarially generated, for example counting social network users or detection of network scanning attacks. This prompts an examination of the performance of the HLL algorithm in the face of adversarial inputs. We show that in such a setting, the HLL algorithm’s estimate of cardinality can be exponentially bad: when an adversary has access to the internals of the HLL algorithm and has some flexibility in choosing what inputs will be recorded, it can manipulate the cardinality estimate to be exponentially smaller than the true cardinality. We study both the original HLL algorithm and a more modern version of it (Ertl, 2017) that is used in Redis. We present experimental results confirming our theoretical analysis. Finally, we consider attack prevention: we show how to modify HLL in a simple way that provably prevents cardinality estimate manipulation attacks.

Note: Updated with additional results and experiments.

Available format(s)
Category
Applications
Publication info
Preprint. MINOR revision.
Contact author(s)
kenny paterson @ inf ethz ch
mathilde raynal @ epfl ch
History
2022-02-21: last of 2 revisions
See all versions
Short URL
https://ia.cr/2021/1139

CC BY

BibTeX

@misc{cryptoeprint:2021/1139,
author = {Kenneth G.  Paterson and Mathilde Raynal},
title = {HyperLogLog: Exponentially Bad in Adversarial Settings},
howpublished = {Cryptology ePrint Archive, Paper 2021/1139},
year = {2021},
note = {\url{https://eprint.iacr.org/2021/1139}},
url = {https://eprint.iacr.org/2021/1139}
}

Note: In order to protect the privacy of readers, eprint.iacr.org does not use cookies or embedded third party content.