Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] pushdown or predicate (7): support bitmap filter #45603

Merged
merged 5 commits into from
May 23, 2024

Conversation

ZiheLiu
Copy link
Contributor

@ZiheLiu ZiheLiu commented May 14, 2024

Why I'm doing:

Support bitmap filter for OR predicates.

What I'm doing:

The status for a predicate to apply bitmap index contains 4 kinds:

  • NOT_USED: cannot apply bitmap index.
  • ALWAYS_TRUE: the predicate always returns true, eg. c1 in (1, 2, 3) and bitmap index also only contains (1, 2, 3).
  • ALWAYS_FALSE: the predicate always returns false, eg. c1 in (1, 2, 3) and bitmap index does not contain any of 1, 2, 3.
  • OK: the predicate can apply bitmap index and does not always return true or false, that is , need to evaluate with bitmap index data.

AND and OR deal with each status in different ways:

  • NOT_USED
    • AND:
      • If there is any non-NOT_USED child predicate, use them to apply bitmap index.
      • If there is not non-NOT_USED child predicate, return NOT_USED.
    • OR:
      • If there is any NOT_USED child predicate, return NOT_USED.
  • ALWAYS_TRUE
    • AND:
      • Erase ALWAYS_TRUE child predicates.
      • If there is not non-ALWAYS_TRUE child predicate, return ALWAYS_TRUE.
    • OR:
      • If there is any ALWAYS_TRUE predicate, return ALWAYS_TRUE.
  • ALWAYS_FALSE
    • AND:
      • If there is any ALWAYS_FALSE predicate, return ALWAYS_FALSE.
    • OR:
      • Erase ALWAYS_FALSE child predicates.
      • If there is not non-ALWAYS_FALSE child predicate, return ALWAYS_FALSE.
  • OK
    • Both AND and OR uses OK child predicates to apply bitmap index.

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

… predicate

Signed-off-by: zihe.liu <ziheliu1024@gmail.com>

bitmap

Signed-off-by: zihe.liu <ziheliu1024@gmail.com>

s

Signed-off-by: zihe.liu <ziheliu1024@gmail.com>

bitmap

Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
@ZiheLiu ZiheLiu requested review from a team as code owners May 14, 2024 11:35
@ZiheLiu ZiheLiu changed the title [Enhancement] pushdown or predicate (8): support bitmap filter [Enhancement] pushdown or predicate (7): support bitmap filter May 14, 2024
@ZiheLiu ZiheLiu force-pushed the feat/main/or_pred_7_bitmap branch from 8b3fb7f to 3c8854a Compare May 20, 2024 08:07
Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
@ZiheLiu ZiheLiu force-pushed the feat/main/or_pred_7_bitmap branch from d769957 to 651a3fd Compare May 20, 2024 12:48
Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
Signed-off-by: ZiheLiu <ziheliu1024@gmail.com>
@StarRocks StarRocks deleted a comment from github-actions bot May 21, 2024
Copy link

[FE Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

[BE Incremental Coverage Report]

pass : 293 / 309 (94.82%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 be/src/storage/column_operator_predicate.h 0 1 00.00% [162]
🔵 be/src/storage/rowset/bitmap_index_evaluator.cpp 224 239 93.72% [103, 124, 167, 184, 249, 250, 251, 259, 277, 290, 297, 312, 313, 436, 437]
🔵 be/src/storage/column_not_in_predicate.cpp 12 12 100.00% []
🔵 be/src/storage/rowset/bitmap_index_evaluator.h 6 6 100.00% []
🔵 be/src/storage/column_null_predicate.cpp 4 4 100.00% []
🔵 be/src/storage/predicate_tree/predicate_tree.cpp 2 2 100.00% []
🔵 be/src/storage/column_in_predicate.cpp 4 4 100.00% []
🔵 be/src/storage/column_predicate.h 1 1 100.00% []
🔵 be/src/storage/column_predicate_cmp.cpp 12 12 100.00% []
🔵 be/src/storage/rowset/segment_iterator.cpp 28 28 100.00% []

@wyb wyb merged commit cc7abe0 into StarRocks:main May 23, 2024
51 checks passed
@ZiheLiu ZiheLiu mentioned this pull request May 23, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants