[SPARK-25084] "distribute by" on multiple columns may lead to codegen issue

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.3.2, 2.4.0
• Component/s: SQL
• Assignee: yucai
• Reporter: yucai
• Created: 10/Aug/18 05:19
• Updated: 13/Aug/18 00:51
• Resolved: 11/Aug/18 13:39

Description

Test Query:

select * from store_sales distribute by (ss_sold_time_sk, ss_item_sk, ss_customer_sk, ss_cdemo_sk, ss_addr_sk, ss_promo_sk) limit 1;

Exception:

Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: One of ', )' expected instead of '['
at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:1435)
at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1497)
at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1494)
at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at org.spark_project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

Wrong Codegen:

/* 131 */ private int computeHashForStruct_1(InternalRow mutableStateArray[0], int value1) {
/* 132 */
/* 133 */
/* 134 */ if (!mutableStateArray[0].isNullAt(5)) {
/* 135 */
/* 136 */ final int element5 = mutableStateArray[0].getInt(5);
/* 137 */ value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element5, value1);
/* 138 */
/* 139 */ }

 

Issue Links

• links to: [Github] Pull Request #22066 (yucai)
• links to: [Github] Pull Request #22067 (LantaoJin)
• links to: [Github] Pull Request #22077 (LantaoJin)