Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25084

"distribute by" on multiple columns may lead to codegen issue

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.3.1
    • 2.3.2, 2.4.0
    • SQL
    • None

    Description

      Test Query:

      select * from store_sales distribute by (ss_sold_time_sk, ss_item_sk, ss_customer_sk, ss_cdemo_sk, ss_addr_sk, ss_promo_sk) limit 1;

      Exception:

      Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 131, Column 67: One of ', )' expected instead of '['
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:1435)
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1497)
      at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1494)
      at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
      at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
      at org.spark_project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

      Wrong Codegen:

      /* 131 */ private int computeHashForStruct_1(InternalRow mutableStateArray[0], int value1) {
      /* 132 */
      /* 133 */
      /* 134 */ if (!mutableStateArray[0].isNullAt(5)) {
      /* 135 */
      /* 136 */ final int element5 = mutableStateArray[0].getInt(5);
      /* 137 */ value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element5, value1);
      /* 138 */
      /* 139 */ }

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yucai yucai
            yucai yucai
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment