[VL] Result mismatch issues Tracker

[zhouyuan]

Backend

VL (Velox)

Bug description

There are several data mismatch issues either related with operator or functions. Some of the fixes are landed in Gluten, and some are in Velox repo.
We will use this issue to track the status as these are critical for production envs.

• complex datatype return wrong value, disabled in Gluten now
• parquet scan + filter pushdown wrongly return "", should return null.
• distinct hash agg + spill returned duplicated keys.
• max_by function return wrong result
• sortBeforeRepartition
• cast(sum(decmial(20,4)), float)
• get_json_object({"dScore":0.0215434648799772}, "$.dScore") (Fixed in Velox)
• cast(string as bigint)
• cast(double as decimal)
• array_size(null)
• round(avg(cast(col as double)), 4) [VL] Results are mismatch with Vanilla Spark when round(avg(cast(col as double)), 4) in release-1.1 #5366
• isNull and isNotNull in filter condition [GLUTEN-5682][VL] Fix incorrect result when isNull & isNotNull coexist in filter #5670
• from_unixtime with overflowed
• date_format ([VL] date_format returns wrong results #5524)
• cast integer as binary ([VL] Casting integer to binary type produces inconsistent result with Spark #5073)

[FelixYBW]

#4678 issue in hashagg

[FelixYBW]

#4587

Currently we disabled all complex data read

[zhouyuan]

#4818

[zhouyuan]

#4872

[kecookier]

#4891

[kecookier]

#4928

[kecookier]

#4930

[kecookier]

#4947

[FelixYBW]

3 issues we met:

1. parquet scan + filter pushdown wrongly return "", should return null. Fixed by Fix read parquet for different encodings across row groups facebookincubator/velox#9129
2. distinct hash agg + spill returned duplicated keys. distinct hash aggregate returned duplicated value if spill happens facebookincubator/velox#9219
3. max_by function return wrong result

[ulysses-you]

2. distinct hash agg + spill returned duplicated keys.

@FelixYBW Has this issue not been fixed by #4443 ?

[FelixYBW]

@FelixYBW Has this issue not been fixed by #4443 ?

No, it's tested from main branch. A new issue

[FelixYBW]

No, it's tested from main branch. A new issue

facebookincubator/velox#9219

[FelixYBW]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

[yma11]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

[NEUpanning]

#5253

[FelixYBW]

#5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

[PHILO-HE]

#5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

@FelixYBW, I will do that. Thanks!

[FelixYBW]

#5248

[kecookier]

#5366

[FelixYBW]

#5366

UPdated desc. thank you. do you know which function (cast, avg, round ) caused the issue?

[FelixYBW]

#5372

[yma11]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

[FelixYBW]

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

[yma11]

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

Yes. I think so. It's caused by the additional projects we added before/after shuffle. The logic of partial/final handle in Velox upstream has no problem. The ideal way is to add struct support for shuffle in Gluten so that we can remove the hack.

[FelixYBW]

@PHILO-HE Any update of the issues here?

[zjuwangg]

#5682

[NEUpanning]

#5701

[PHILO-HE]

@PHILO-HE Any update of the issues here?

@FelixYBW, Some were actually fixed. Just updated the list. Will fix or seek help to fix other issues.