Paper 2025/1173

The Effectiveness of Differential Privacy in Real-world Settings: A Metrics-based Framework to help Practitioners Visualise and Evaluate $\varepsilon$

Abstract

Differential privacy (DP) has emerged as a preferred solution for private data analysis, having been adopted by several leading Internet companies. DP is a privacy-preserving mechanism that protects against re-identification of individuals. \emph{Privacy budget} $\varepsilon$ determines the trade-off between privacy and utility. Currently, there are no widely accepted methodologies for selecting $\varepsilon$ and choosing the best DP mechanism that offers an optimal trade-off between privacy and utility. We propose using a novel set of metrics and an easy-to-implement, step-by-step framework to facilitate DP deployment and guide the selection of $\varepsilon$ based on the desired accuracy vs utility trade-off. To address this gap, we perform experiments using three real-world datasets, aiming to identify the optimal parameters . The analysis focuses on the commonly used queries for each dataset. Based on our experiment results, we observe that using our metric and framework, one can analyse noise distribution of multiple queries, and choose the suitable $\varepsilon$ and the DP mechanism for achieving a balance between privacy and utility. Additionally, we show that the optimal $\varepsilon$ depends on the particular query, desired accuracy and context in which DP is implemented, which suggests that an arbitrary a prior selection of $\varepsilon$ cannot provide adequate results. Our framework prioritises the visualisation of key metrics in the DP analysis as a function of $\varepsilon$ and mechanism. Enabling the adoption easy for a wider audience without deep knowledge, providing a road map for DP implementation.