Optimize (left outer) (anti) semi join with other conditions #8339

gengliqi · 2023-11-08T17:59:46Z

What problem does this PR solve?

Issue Number: close #8262

Problem Summary:
See #8262

What is changed and how it works?

semi join with other conditions can finish quickly in some cases. It does not need to combine all matched rows and then calculate the other conditions.
The size of block in the process of calculating other conditions is fixed even if matched rows are huge. No OOM will happen when using semi join.
semi join with no other condition is also optimized in this PR. In the previous implementation, semi join added matched row columns in the right table. Actually, these columns are useless and will be removed soon. In this PR, these columns will not be added anymore.

// Before + TPC-H 1
mysql> select count(*) from customer a where exists (select 1 from customer b where a.C_NATIONKEY=b.C_NATIONKEY and a.C_PHONE > b.C_PHONE);
+----------+
| count(*) |
+----------+
|   149975 |
+----------+
1 row in set (16.04 sec)

// This PR + TPC-H 1
mysql> select count(*) from customer a where exists (select 1 from customer b where a.C_NATIONKEY=b.C_NATIONKEY and a.C_PHONE > b.C_PHONE);
+----------+
| count(*) |
+----------+
|   149975 |
+----------+
1 row in set (0.08 sec)

// Before + TPC-H 100
mysql> select count(*) from customer a where exists (select 1 from customer b where a.C_NATIONKEY=b.C_NATIONKEY and a.C_PHONE > b.C_PHONE);
ERROR 1105 (HY000): rpc error: code = Canceled desc = context canceled

// This PR + TPC-H 100
mysql> select count(*) from customer a where exists (select 1 from customer b where a.C_NATIONKEY=b.C_NATIONKEY and a.C_PHONE > b.C_PHONE);
+----------+
| count(*) |
+----------+
| 14999975 |
+----------+
1 row in set (0.89 sec)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Optimize (left outer) (anti) semi join with other conditions

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-08T18:08:40Z

/run-all-tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-09T07:57:25Z

/run-all-tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-09T08:08:35Z

/run-all-tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-09T09:59:40Z

/run-all-tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-14T07:00:49Z

/run-all-tests

gengliqi · 2023-11-14T07:43:40Z

/run-integration-test

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-15T16:28:07Z

/rebuild

dbms/src/Interpreters/Join.cpp

windtalker · 2023-11-16T08:40:20Z

dbms/src/Interpreters/Join.cpp

+        block.insert(src_column);
+    }
+
+    if constexpr (STRICTNESS == ASTTableJoin::Strictness::All)


STRICTNESS == ASTTableJoin::Strictness::All means has other condition?

windtalker · 2023-11-16T08:45:42Z

dbms/src/Interpreters/Join.cpp

+    if constexpr (KIND == ASTTableJoin::Kind::LeftOuterSemi || KIND == ASTTableJoin::Kind::LeftOuterAnti)
+    {
+        auto * left_semi_column = typeid_cast<ColumnNullable *>(added_columns[right_columns - 1].get());
+        left_semi_column_data = &typeid_cast<ColumnVector<Int8> &>(left_semi_column->getNestedColumn()).getData();


reserve for left_semi_column_data?

Nice catch!

windtalker · 2023-11-16T08:51:26Z

dbms/src/Interpreters/Join.cpp

+        else
+        {
+            auto result = res[i].getResult();
+            if constexpr (KIND == ASTTableJoin::Kind::Semi || KIND == ASTTableJoin::Kind::Anti)


Looks like for Semi and Anti join, the code branch of Any/All strictness can be merged into one if you define operator bool() on the result as result == SemiJoinResultType::TRUE_VALUE

Good idea! I use the function isTrueSemiJoinResult because result is an enum or bool type.

ti-chi-bot · 2023-11-22T20:11:54Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-11-29T12:29:04Z

/run-all-tests

windtalker · 2023-11-30T07:26:12Z

Need to set probe_cache_column_threshold to 0 here for semi join

tiflash/dbms/src/Interpreters/Join.cpp

Lines 187 to 188 in e5eaf31

    
           if (isNullAwareSemiFamily(kind)) 
        
               probe_cache_column_threshold = 0;

Signed-off-by: gengliqi <gengliqiii@gmail.com>

windtalker

LGTM

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-12-05T09:43:52Z

/run-all-tests

yibin87

Others LGTM

dbms/src/Interpreters/Join.cpp

ti-chi-bot · 2023-12-06T02:48:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, yibin87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [windtalker,yibin87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-12-06T02:48:10Z

[LGTM Timeline notifier]

Timeline:

2023-11-30 08:00:56.824986398 +0000 UTC m=+1082485.490212593: ☑️ agreed by windtalker.
2023-12-06 02:48:09.681985647 +0000 UTC m=+1582118.347211842: ☑️ agreed by yibin87.

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-12-06T06:31:21Z

/run-all-tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi · 2023-12-06T07:13:21Z

/run-all-tests

gengliqi · 2023-12-06T07:35:56Z

/run-all-tests

gengliqi · 2023-12-06T08:18:27Z

/run-all-tests

gengliqi · 2023-12-06T08:59:51Z

/run-unit-test

gengliqi added 6 commits November 2, 2023 17:01

fix tests

4560d99

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Merge branch 'master' into add-semi-join-type

946d856

merge master

802344a

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Merge remote-tracking branch 'upstream/master' into refine-semi-join

fbc9d57

optimize test

e088987

Signed-off-by: gengliqi <gengliqiii@gmail.com>

optimize semi join

d058df3

Signed-off-by: gengliqi <gengliqiii@gmail.com>

ti-chi-bot bot added release-note-none size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 8, 2023

format

518efbb

Signed-off-by: gengliqi <gengliqiii@gmail.com>

make Static Analysis happy

ba3488a

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi changed the title ~~Rewrite (left outer) (anti) semi join to make it faster~~ Optimize (left outer) (anti) semi join with other conditions Nov 9, 2023

format

68ae758

Signed-off-by: gengliqi <gengliqiii@gmail.com>

fix test

33f6d1a

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi requested review from windtalker and yibin87 November 9, 2023 10:52

gengliqi force-pushed the refine-semi-join branch from 797a3b8 to 33f6d1a Compare November 10, 2023 03:51

remove useless code

8c450d3

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi added 2 commits November 15, 2023 18:51

add tests

f538cf2

Signed-off-by: gengliqi <gengliqiii@gmail.com>

format

07ad544

Signed-off-by: gengliqi <gengliqiii@gmail.com>

windtalker reviewed Nov 16, 2023

View reviewed changes

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 22, 2023

address comments

d315135

Signed-off-by: gengliqi <gengliqiii@gmail.com>

address comments

91d1f54

Signed-off-by: gengliqi <gengliqiii@gmail.com>

windtalker approved these changes Nov 30, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm approved labels Nov 30, 2023

gengliqi added 2 commits November 30, 2023 16:25

make clang-tidy happy

2a1ae76

Signed-off-by: gengliqi <gengliqiii@gmail.com>

update

e25d317

Signed-off-by: gengliqi <gengliqiii@gmail.com>

gengliqi force-pushed the refine-semi-join branch from 1ea3676 to e25d317 Compare December 5, 2023 08:02

make clang-tidy happy

eabebd2

Signed-off-by: gengliqi <gengliqiii@gmail.com>

ti-chi-bot bot added release-note and removed release-note-none labels Dec 5, 2023

yibin87 approved these changes Dec 6, 2023

View reviewed changes

dbms/src/Interpreters/Join.cpp Outdated Show resolved Hide resolved

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm labels Dec 6, 2023

gengliqi added 2 commits December 6, 2023 14:30

address comments

19bad98

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Merge branch 'master' into refine-semi-join

6f456f4

gengliqi added 2 commits December 6, 2023 15:04

make clang-tidy happy

31c1728

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Merge branch 'master' into refine-semi-join

c932d1e

gengliqi removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 6, 2023

ti-chi-bot bot merged commit 88f9912 into pingcap:master Dec 6, 2023
6 checks passed

gengliqi deleted the refine-semi-join branch December 6, 2023 09:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize (left outer) (anti) semi join with other conditions #8339

Optimize (left outer) (anti) semi join with other conditions #8339

gengliqi commented Nov 8, 2023 •

edited

gengliqi commented Nov 8, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 14, 2023

gengliqi commented Nov 14, 2023

gengliqi commented Nov 15, 2023

windtalker Nov 16, 2023

gengliqi Nov 28, 2023

windtalker Nov 16, 2023

gengliqi Nov 28, 2023

windtalker Nov 16, 2023

gengliqi Nov 28, 2023

ti-chi-bot bot commented Nov 22, 2023

gengliqi commented Nov 29, 2023

windtalker commented Nov 30, 2023

windtalker left a comment

gengliqi commented Dec 5, 2023

yibin87 left a comment

ti-chi-bot bot commented Dec 6, 2023

ti-chi-bot bot commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

Optimize (left outer) (anti) semi join with other conditions #8339

Optimize (left outer) (anti) semi join with other conditions #8339

Conversation

gengliqi commented Nov 8, 2023 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

gengliqi commented Nov 8, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 9, 2023

gengliqi commented Nov 14, 2023

gengliqi commented Nov 14, 2023

gengliqi commented Nov 15, 2023

windtalker Nov 16, 2023

Choose a reason for hiding this comment

gengliqi Nov 28, 2023

Choose a reason for hiding this comment

windtalker Nov 16, 2023

Choose a reason for hiding this comment

gengliqi Nov 28, 2023

Choose a reason for hiding this comment

windtalker Nov 16, 2023

Choose a reason for hiding this comment

gengliqi Nov 28, 2023

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 22, 2023

gengliqi commented Nov 29, 2023

windtalker commented Nov 30, 2023

windtalker left a comment

Choose a reason for hiding this comment

gengliqi commented Dec 5, 2023

yibin87 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 6, 2023

ti-chi-bot bot commented Dec 6, 2023

[LGTM Timeline notifier]

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Dec 6, 2023

gengliqi commented Nov 8, 2023 •

edited