[BugFix] fix stale histogram lead to unexpected stats #45614

packy92 · 2024-05-14T13:02:49Z

Why I'm doing:

Column histogram may not updated after the first collection.
If use the stale histogram to estimate row number may have a risk of divide zero exception because the estimated histogram may return 0 row count(a empty bucket and empty mcv).

What I'm doing:

If the predicate range doesn't overlap with the histogram, we use the min/max value instead of histogram to estimate statistics.

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: packy92 <wangchao@starrocks.com>

starrocks-cr · 2024-05-14T13:04:15Z

...src/main/java/com/starrocks/sql/optimizer/statistics/BinaryPredicateStatisticCalculator.java

+            return Optional.empty();
+        }
+
+        return Optional.of(new Histogram(bucketList, estimatedMCV));
    }

    public static ColumnStatistic estimateColumnStatisticsWithHistogram(ColumnStatistic columnStatistic,


The most risky bug in this code is:
There is a potential NullPointerException or unhandled empty Optional being returned from updateHistWithLessThan and updateHistWithGreaterThan, which are used without checking whether the Optional contains a value before calling .get() method. This might lead to NoSuchElementException if the Optional is empty.

You can modify the code like this:
Inside both estimateColumnLessThanConstant and estimateColumnGreaterThanConstant methods, replace the lines where Histogram estimatedHistogram = hist.get(); is used with the following:

if (!hist.isPresent()) { // Handle the case where histogram is not present, for example: return statistics; // Returning unchanged statistics as we cannot estimate } Histogram estimatedHistogram = hist.get();

This ensures that you handle the case where hist might be empty properly, rather than directly calling .get() on an Optional that could potentially be empty, avoiding NoSuchElementException.

Signed-off-by: packy92 <wangchao@starrocks.com>

sonarcloud · 2024-05-15T02:20:05Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.3% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-05-15T04:28:44Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-05-15T04:28:48Z

[FE Incremental Coverage Report]

✅ pass : 31 / 34 (91.18%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/statistics/BinaryPredicateStatisticCalculator.java	27	30	90.00%	[397, 440, 518]
🔵	com/starrocks/sql/optimizer/statistics/Histogram.java	4	4	100.00%	[]

github-actions · 2024-05-15T05:41:10Z

@Mergifyio backport branch-3.3

github-actions · 2024-05-15T05:41:12Z

@Mergifyio backport branch-3.2

github-actions · 2024-05-15T05:41:13Z

@Mergifyio backport branch-3.1

github-actions · 2024-05-15T05:41:15Z

@Mergifyio backport branch-3.0

github-actions · 2024-05-15T05:41:16Z

@Mergifyio backport branch-2.5

mergify · 2024-05-15T05:41:16Z

backport branch-3.3

✅ Backports have been created

#45642 [BugFix] fix stale histogram lead to unexpected stats (backport #45614) has been created for branch branch-3.3

mergify · 2024-05-15T05:41:17Z

backport branch-3.2

✅ Backports have been created

#45643 [BugFix] fix stale histogram lead to unexpected stats (backport #45614) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-05-15T05:41:21Z

backport branch-3.1

✅ Backports have been created

#45644 [BugFix] fix stale histogram lead to unexpected stats (backport #45614) has been created for branch branch-3.1 but encountered conflicts

mergify · 2024-05-15T05:41:23Z

backport branch-3.0

✅ Backports have been created

#45645 [BugFix] fix stale histogram lead to unexpected stats (backport #45614) has been created for branch branch-3.0 but encountered conflicts

mergify · 2024-05-15T05:41:24Z

backport branch-2.5

✅ Backports have been created

#45646 [BugFix] fix stale histogram lead to unexpected stats (backport #45614) has been created for branch branch-2.5 but encountered conflicts

Signed-off-by: packy92 <wangchao@starrocks.com> (cherry picked from commit 1e5626e)

Signed-off-by: packy92 <wangchao@starrocks.com> (cherry picked from commit 1e5626e) # Conflicts: # fe/fe-core/src/test/resources/sql/tpch-histogram-cost/q19.sql

Signed-off-by: packy92 <wangchao@starrocks.com> (cherry picked from commit 1e5626e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/Histogram.java # fe/fe-core/src/test/resources/sql/tpch-histogram-cost/q19.sql

Signed-off-by: packy92 <wangchao@starrocks.com> (cherry picked from commit 1e5626e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/BinaryPredicateStatisticCalculator.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/Histogram.java # fe/fe-core/src/test/resources/sql/tpch-histogram-cost/q19.sql

… (#45642) Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com>

… (#45643) Signed-off-by: packy92 <wangchao@starrocks.com> Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com> Co-authored-by: packy92 <wangchao@starrocks.com>

… (#45644) Signed-off-by: packy92 <wangchao@starrocks.com> Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com> Co-authored-by: packy92 <wangchao@starrocks.com>

… (#45645) Signed-off-by: packy92 <wangchao@starrocks.com> Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com> Co-authored-by: packy92 <wangchao@starrocks.com>

… (#45646) Signed-off-by: packy92 <wangchao@starrocks.com> Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com> Co-authored-by: packy92 <wangchao@starrocks.com>

Signed-off-by: packy92 <wangchao@starrocks.com>

[BugFix] fix stale histogram lead to unexpected stats

7ff4204

Signed-off-by: packy92 <wangchao@starrocks.com>

packy92 requested a review from a team as a code owner May 14, 2024 13:02

mergify bot assigned packy92 May 14, 2024

starrocks-cr bot reviewed May 14, 2024

View reviewed changes

github-actions bot added 3.3 3.2 3.1 3.0 2.5 labels May 14, 2024

kangkaisen previously approved these changes May 14, 2024

View reviewed changes

fix style

4fe4614

Signed-off-by: packy92 <wangchao@starrocks.com>

packy92 dismissed kangkaisen’s stale review via 4fe4614 May 14, 2024 13:13

fix ut

1d35e41

Signed-off-by: packy92 <wangchao@starrocks.com>

LiShuMing approved these changes May 15, 2024

View reviewed changes

kangkaisen approved these changes May 15, 2024

View reviewed changes

kangkaisen merged commit 1e5626e into StarRocks:main May 15, 2024
47 checks passed

github-actions bot removed the 3.3 label May 15, 2024

github-actions bot removed the 3.2 label May 15, 2024

github-actions bot removed the 3.1 label May 15, 2024

github-actions bot removed the 3.0 label May 15, 2024

github-actions bot removed the 2.5 label May 15, 2024

mergify bot pushed a commit that referenced this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (#45614)

ad37d3b

Signed-off-by: packy92 <wangchao@starrocks.com> (cherry picked from commit 1e5626e)

mergify bot mentioned this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614) #45642

Merged

42 tasks

mergify bot mentioned this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614) #45643

Merged

42 tasks

mergify bot mentioned this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614) #45644

Merged

42 tasks

mergify bot mentioned this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614) #45645

Merged

42 tasks

mergify bot mentioned this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614) #45646

Merged

42 tasks

wanpengfei-git pushed a commit that referenced this pull request May 15, 2024

[BugFix] fix stale histogram lead to unexpected stats (backport #45614)…

595e1b5

… (#45642) Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com>

github-actions bot added the 3.3-merged label May 15, 2024

github-actions bot added the 3.2-merged label May 15, 2024

github-actions bot added the 3.1-merged label May 15, 2024

github-actions bot added the 3.0-merged label May 15, 2024

github-actions bot added the 2.5-merged label May 16, 2024

node pushed a commit to vivo/starrocks that referenced this pull request May 17, 2024

[BugFix] fix stale histogram lead to unexpected stats (StarRocks#45614)

653f709

Signed-off-by: packy92 <wangchao@starrocks.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] fix stale histogram lead to unexpected stats #45614

[BugFix] fix stale histogram lead to unexpected stats #45614

packy92 commented May 14, 2024 •

edited

starrocks-cr bot May 14, 2024

sonarcloud bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

[BugFix] fix stale histogram lead to unexpected stats #45614

[BugFix] fix stale histogram lead to unexpected stats #45614

Conversation

packy92 commented May 14, 2024 • edited

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot May 14, 2024

Choose a reason for hiding this comment

sonarcloud bot commented May 15, 2024

Quality Gate passed

github-actions bot commented May 15, 2024

[BE Incremental Coverage Report]

github-actions bot commented May 15, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 15, 2024

mergify bot commented May 15, 2024 • edited

✅ Backports have been created

mergify bot commented May 15, 2024 • edited

✅ Backports have been created

mergify bot commented May 15, 2024 • edited

✅ Backports have been created

mergify bot commented May 15, 2024 • edited

✅ Backports have been created

mergify bot commented May 15, 2024 • edited

✅ Backports have been created

packy92 commented May 14, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited

mergify bot commented May 15, 2024 •

edited