[VL] Add a config to ignore fallback cost for scan #5617
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
PHILO-HE
force-pushed
the
fallback-ignore-case
branch
from
64b001a
to
9c56d68
Compare
|
Run Gluten Clickhouse CI |
| @@ -208,7 +208,7 @@ class GlutenConfig(conf: SQLConf) extends Logging { | |||
|
|
|||
| def queryFallbackThreshold: Int = conf.getConf(COLUMNAR_QUERY_FALLBACK_THRESHOLD) | |||
|
|
|||
| def fallbackIgnoreRowToColumnar: Boolean = conf.getConf(COLUMNAR_FALLBACK_IGNORE_ROW_TO_COLUMNAR) | |||
| def ignoreScanFallbackCost: Boolean = conf.getConf(COLUMNAR_IGNORE_SCAN_FALLBACK_COST) | |||
There was a problem hiding this comment.
Is the Row_to_column still used? I think it's only used by table scan
There was a problem hiding this comment.
The config for ignoring RowToColumnar looks useless. So I'm removing it. It needs some discussion with our customer.
There was a problem hiding this comment.
I think it's still useful, fallbackIgnoreRowToColumnar allows convert shuffle to columnar shuffle
There was a problem hiding this comment.
@ulysses-you, I feel it is more clear to keep considering Row2Columnar, because we are calculating all transition cost (C2R/R2C) in the current evaluation logic. The Row2Columnar doesn't only exist prior to columnar shuffle. So it may be not that good to keep fallbackIgnoreRowToColumnar just for the purpose of using columnar shuffle. If we prefer using columnar shuffle, maybe we can add some dedicate code to realize the purpose.
|
Run Gluten Clickhouse CI |
PHILO-HE
force-pushed
the
fallback-ignore-case
branch
from
b7bfa2e
to
2c199c2
Compare
|
Run Gluten Clickhouse CI |
PHILO-HE
force-pushed
the
fallback-ignore-case
branch
from
2c199c2
to
b5992e6
Compare
|
Run Gluten Clickhouse CI |
PHILO-HE
force-pushed
the
fallback-ignore-case
branch
from
b5992e6
to
690c44d
Compare
|
Run Gluten Clickhouse CI |
2 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
PHILO-HE
force-pushed
the
fallback-ignore-case
branch
from
95a7a73
to
475f998
Compare
|
Run Gluten Clickhouse CI |
It's common that the file format isn't supported by Gluten like Hive Row format which won't be supported in near future, but the rest of the operators are already supported by Gluten. It's high possibility we can still get perf gain from the rest operators - the R2C cost. So let's add a specific config for the scan only