PROBES: A Bernstein Simplex Approach to Differentially Private OLS Estimation

Differentially private ordinary least squares regression on bounded continuous data is dominated by a single design tension: each sufficient statistic $\{S_{x_j}, S_y, S_{x_j x_k}, S_{x_j y}\}$ is conventionally treated as an independent query, forcing the analyst to either split the privacy budget $\varepsilon$ across many releases or compose a long sequence of noisy mechanisms. For $p$-dimensional multivariate linear regression, this implies estimating $\Theta(p^2)$ quantities, thereby inflating per-statistic noise by a factor of $\Theta(p^2)$.

We introduce PROBES (PRivate O}rdinary least squares via BErnstein Simplex), a pure $\varepsilon$-DP mechanism that exploits the common algebraic structure shared across the sufficient statistics, releases a single Laplace vector at the full unsplit budget $\varepsilon$, and recovers every sufficient statistic for multivariate linear regression on $p$ bounded features by post-processing. The construction rests on the Bernstein simplex embedding: each record is mapped to a non-negative vector of dimension $3^p \times 2$ that sums to exactly 1, giving a global $\ell_1$-sensitivity of 1 regardless of $p$.

We also present Ada-PROBES, which retains the single-shot Bernstein release of PROBES and recovers coefficients via a noise-calibrated ridge solve that stabilizes matrix inversion at no extra privacy cost. Experiments on six real-world regression benchmarks show that Ada-PROBES achieves test RMSE within 1% of non-private OLS for $p \le 4$ on large-sample datasets at $\varepsilon = 1$, while DiffPrivLib collapses numerically and AdaSSP/BinAgg incur noticeably higher error in the same low-$p$ regime.

For $p \ge 6$, we introduce S-PROBES, a scalable Steiner-block based pure $\varepsilon$-DP extension that preserves the single-release structure per-block and remains competitive at relatively higher values of $p$ with the $(\varepsilon, \delta)$-DP methods AdaSSP and BinAgg (which employ Gaussian noise based on $\ell_2$-sensitivity).