New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-5696] Add preprojection support for ArrowEvalPythonExec #5697
Conversation
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
@jinchengchenghh @zhouyuan please help take a review. Thanks. |
@@ -738,5 +739,7 @@ trait SparkPlanExecApi { | |||
|
|||
def genPostProjectForGenerate(generate: GenerateExec): SparkPlan | |||
|
|||
def genPreProjectForArrowEvalPythonExec(arrowEvalPythonExec: ArrowEvalPythonExec): SparkPlan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can default return arrowEvalPythonExec
, and remove the change in CH backend
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated.
Run Gluten Clickhouse CI |
val argOffsets = inputs.map { | ||
input => | ||
input.map { | ||
e => | ||
if (allInputs.exists(_.semanticEquals(e))) { | ||
allInputs.indexWhere(_.semanticEquals(e)) | ||
} else { | ||
if (!e.isInstanceOf[AttributeReference]) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
else if (!e.isInstanceOf[AttributeReference]) {}
else {}
...ds-velox/src/main/scala/org/apache/gluten/execution/python/ColumnarArrowEvalPythonExec.scala
Show resolved
Hide resolved
// flatten all the arguments | ||
val allInputs = new ArrayBuffer[Expression] | ||
for (input <- inputs) { | ||
input.map { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
input.filter(allInputs.exists(_.semanticEquals(e))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line is not to filter the inputs but add the matched expression in allInputs
and replace it using faked Attribute
.
backends-velox/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
Outdated
Show resolved
Hide resolved
backends-velox/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
Outdated
Show resolved
Hide resolved
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
What changes were proposed in this pull request?
Add preprojection support for ArrowEvalPythonExec
Fixes: #5696)
How was this patch tested?
New UTs added