Documented noisy aggregate functions in new page #22715

jonhehir · 2024-05-10T18:31:59Z

Description

Moved documentation for noisy_count_gaussian and related functions from the Aggregate Functions page to a new Noisy Aggregate Functions page. Also added documentation of the SFM sketch type and functions, such as noisy_approx_set_sfm.

Motivation and Context

This is the documentation promised back in #21290.

Impact

Docs only

Test Plan

Docs build and render nicely.

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Add documentation for noisy aggregate functions at :doc:`/functions/noisy`, including :func:`noisy_approx_distinct_sfm` and :func:`noisy_approx_set_sfm` (:pr:`21290`, :pr:`22715`)

elharo · 2024-05-11T11:12:30Z

presto-docs/src/main/sphinx/functions/noisy.rst

+Noisy Aggregate Functions
+=========================
+
+Noisy aggregate functions are aggregate functions that provide random, noisy


are aggregate functions --> are functions

elharo · 2024-05-11T11:14:07Z

presto-docs/src/main/sphinx/functions/noisy.rst

+    mechanisms, it is important to note that neither the values returned by
+    these functions nor the query results that incorporate these functions
+    are differentially private in general. See `Limitations`_ below for more
+    details. Users who wish to support a strong privacy guarantee should


I'd delete " Users who wish to support a strong privacy guarantee should
discuss with a suitable technical expert first."

I'm going to vote to leave this sentence, as we want to be abundantly clear about both the limitations and their practical implications.

elharo · 2024-05-11T11:15:32Z

presto-docs/src/main/sphinx/functions/noisy.rst

+
+.. note::
+
+    Unlike :func:`count`, this function will return ``NULL`` when the (true) count of ``x`` is 0.


will return --> returns

elharo · 2024-05-11T11:16:34Z

presto-docs/src/main/sphinx/functions/noisy.rst

+
+.. function:: noisy_count_gaussian(x, noise_scale[, random_seed]) -> bigint
+
+    Counts the non-``NULL`` values in ``x`` and then adds random Gaussian noise with 0


What's x? (a column I assume)

Indeed, I've renamed all these from x to col, which is hopefully more clear.

elharo · 2024-05-11T11:16:55Z

presto-docs/src/main/sphinx/functions/noisy.rst

+
+    Distinct counting can be performed using ``noisy_count_gaussian(DISTINCT x, ...)``, or with
+    :func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian` will
+    return more accurate results but at a larger computational cost.


will return __> returns

elharo · 2024-05-11T11:22:30Z

presto-docs/src/main/sphinx/functions/noisy.rst

+.. function:: noisy_avg_gaussian(x, noise_scale[, random_seed]) -> double
+    :noindex:
+
+    Calculates the average (arithmetic mean) of all the input values in ``x`` and then adds


"adds a normally distributed random double value with 0 mean..."

elharo · 2024-05-11T11:23:07Z

presto-docs/src/main/sphinx/functions/noisy.rst

+
+.. note::
+
+    Unlike :func:`approx_set`, this function will return ``NULL`` when ``x`` is empty.


will return --> returns

elharo · 2024-05-11T11:23:26Z

presto-docs/src/main/sphinx/functions/noisy.rst

+.. note::
+
+    Unlike :func:`approx_set`, this function will return ``NULL`` when ``x`` is empty.
+    If this behavior is undesirable, :func:`coalesce` with :func:`noisy_empty_approx_set_sfm`.


sentence fragment

elharo · 2024-05-11T11:23:38Z

presto-docs/src/main/sphinx/functions/noisy.rst

+
+.. note::
+
+    Unlike :func:`approx_distinct`, this function will return ``NULL`` when ``x`` is empty.


elharo · 2024-05-11T11:25:01Z

presto-docs/src/main/sphinx/functions/noisy.rst

+    Libraries and How to Fix It. In *Proceedings of the 2022 ACM SIGSAC Conference
+    on Computer and Communications Security* (pp. 471-484).
+
+.. [Hehir2023] Hehir, J., Ting, D., & Cormode, G. (2023). Sketch-Flip-Merge:


https://arxiv.org/abs/2302.02056

I've gone ahead and linked all the references.

steveburnett

Nice work, thanks! I had a suggestion I couldn't make inline so I'll mention it here.

Similar to the table of contents that I suggest adding to noisy.rst, aggregate.rst could benefit from a table of contents even more due to its length. Could you add a table of contents to aggregate.rst, the same as I added a comment in noisy.rst about?

If you do so, you could add a heading to aggregate.rst for Noisy Aggregate Functions, and underneath that header link to the new page with "See :doc:noisy. " underneath it, to help the reader know that there are aggregate functions that are not on the Aggregate Functions page.

Screenshot of aggregate.rst in local build with screenshot.

Screenshot of local build for consideration.

presto-docs/src/main/sphinx/language/types.rst

presto-docs/src/main/sphinx/functions.rst

presto-docs/src/main/sphinx/functions/noisy.rst

steveburnett · 2024-05-13T17:50:09Z

presto-docs/src/main/sphinx/functions/noisy.rst

+    Unlike :func:`count`, this function returns ``NULL`` when the (true) count of ``col`` is 0.
+
+    Distinct counting can be performed using ``noisy_count_gaussian(DISTINCT col, ...)``, or with
+    :func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian` will


Suggested change

:func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian` will

:func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian`

presto-docs/src/main/sphinx/functions/noisy.rst

jonhehir · 2024-05-13T21:58:06Z

Thank you, @steveburnett! I appreciate the edits and reStructuredText pointers. I think I've hit your full wish list. Let me know if all looks good. 😄

Moved documentation for noisy_count_gaussian and related functions from the Aggregate Functions page to a new Noisy Aggregate Functions page. Also added documentation of the SFM sketch type and functions, such as noisy_approx_set_sfm.

steveburnett

LGTM! (docs)

Pull updated branch, new local build of docs, verified changes. New review of the documents as a whole, everything looks good. Thanks!

jonhehir requested review from steveburnett and a team as code owners May 10, 2024 18:32

jonhehir requested a review from presto-oss May 10, 2024 18:32

github-actions bot added the docs label May 10, 2024

elharo reviewed May 11, 2024

View reviewed changes

jonhehir force-pushed the noisy-agg-docs branch from 2fb77d0 to f65cdd7 Compare May 13, 2024 16:16

jonhehir requested a review from elharo May 13, 2024 16:50

steveburnett reviewed May 13, 2024

View reviewed changes

jonhehir requested a review from steveburnett May 13, 2024 21:58

Documented noisy aggregate functions in new page

19c7e02

Moved documentation for noisy_count_gaussian and related functions from the Aggregate Functions page to a new Noisy Aggregate Functions page. Also added documentation of the SFM sketch type and functions, such as noisy_approx_set_sfm.

jonhehir force-pushed the noisy-agg-docs branch from f65cdd7 to 19c7e02 Compare May 13, 2024 21:59

steveburnett approved these changes May 14, 2024

View reviewed changes

jonhehir merged commit d8376e1 into prestodb:master May 15, 2024
57 checks passed

jonhehir mentioned this pull request May 20, 2024

[docs] document noisy_approx_set_sfm_from_index_and_zeros #22508

Closed

fazalmajid mentioned this pull request May 21, 2024

added docs for noisy_approx_set_sfm_from_index_and_zeros #22799

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documented noisy aggregate functions in new page #22715

Documented noisy aggregate functions in new page #22715

jonhehir commented May 10, 2024 •

edited

elharo May 11, 2024

elharo May 11, 2024

jonhehir May 13, 2024

elharo May 11, 2024

elharo May 11, 2024

jonhehir May 13, 2024

elharo May 11, 2024

elharo May 11, 2024

elharo May 11, 2024

elharo May 11, 2024

elharo May 11, 2024

elharo May 11, 2024

jonhehir May 13, 2024

steveburnett left a comment

steveburnett May 13, 2024

jonhehir commented May 13, 2024

steveburnett left a comment


		.. note::

		Unlike :func:`count`, this function will return ``NULL`` when the (true) count of ``x`` is 0.


		.. function:: noisy_count_gaussian(x, noise_scale[, random_seed]) -> bigint

		Counts the non-``NULL`` values in ``x`` and then adds random Gaussian noise with 0


		.. note::

		Unlike :func:`approx_set`, this function will return ``NULL`` when ``x`` is empty.


		.. note::

		Unlike :func:`approx_distinct`, this function will return ``NULL`` when ``x`` is empty.

	:func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian` will
	:func:`noisy_approx_distinct_sfm`. Generally speaking, :func:`noisy_count_gaussian`

Documented noisy aggregate functions in new page #22715

Documented noisy aggregate functions in new page #22715

Conversation

jonhehir commented May 10, 2024 • edited

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonhehir commented May 13, 2024

steveburnett left a comment

Choose a reason for hiding this comment

jonhehir commented May 10, 2024 •

edited