Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25084

"distribute by" on multiple columns may lead to codegen issue

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.1
    • Fix Version/s: 2.3.2, 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Test Query:

      select * from store_sales distribute by (ss_sold_time_sk, ss_item_sk, ss_customer_sk, ss_cdemo_sk, ss_addr_sk, ss_promo_sk) limit 1;

      Exception:

      Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: One of ', )' expected instead of '['
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:1435)
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1497)
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1494)
      at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
      at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
      at org.spark_project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

      Wrong Codegen:

      /* 131 */ private int computeHashForStruct_1(InternalRow mutableStateArray[0], int value1) {
      /* 132 */
      /* 133 */
      /* 134 */ if (!mutableStateArray[0].isNullAt(5)) {
      /* 135 */
      /* 136 */ final int element5 = mutableStateArray[0].getInt(5);
      /* 137 */ value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element5, value1);
      /* 138 */
      /* 139 */ }

       

        Attachments

          Activity

            People

            • Assignee:
              yucai yucai
              Reporter:
              yucai yucai
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: