[SPARK-46295] TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.2, 3.5.0, 4.0.0
• Component/s: SQL
• Labels: correctness
• Assignee: Unassigned
• Reporter: Kazuyuki Tanimura
• Created: 07/Dec/23 00:32
• Updated: 07/Dec/23 08:38

Description

TPCDSQueryTestSuite fails for q39a and a39b with broadcastHashJoinConf and shuffledHashJoinConf. It works fine with sortMergeJoinConf

SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly *TPCDSQueryTestSuite -- -z q39a"

[info] - q39a *** FAILED *** (19 seconds, 139 milliseconds)
[info]   java.lang.Exception: Expected "...25	1.022382911080458[8 ..." but got "...25	1.022382911080458[5 ..."

SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly *TPCDSQueryTestSuite -- -z q39b"

[info] - q39b *** FAILED *** (19 seconds, 351 milliseconds)
[info]   java.lang.Exception: Expected "...34	1.563403519178623[3	3	10427	2	381.25	1.0623056061004696
[info] 3	3315	1	271.75	1.555976998814345	3	3315	2	393.75	1.0196319345405949
[info] 3	3393	1	260.0	1.5009563026568116	3	3393	2	470.25	1.129275872154205
[info] 4	16211	1	257.6666666666667	1.6381074811154002]	4	16211	2	352.25	1....", but got "...34	1.563403519178623[5	3	10427	2	381.25	1.0623056061004696
[info] 3	3315	1	271.75	1.555976998814345	3	3315	2	393.75	1.0196319345405949
[info] 3	3393	1	260.0	1.5009563026568118	3	3393	2	470.25	1.129275872154205
[info] 4	16211	1	257.6666666666667	1.6381074811154]	4	16211	2	352.25	1...." Result did not match