Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9132 CBO: Calcite Operator To Hive Operator (Calcite Return Path)
  3. HIVE-10455

CBO (Calcite Return Path): Different data types at Reducer before JoinOp

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • CBO
    • None

    Description

      The following error occured for cbo_subq_not_in.q

      java.lang.Exception: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x128x0x0x1 with properties {columns=reducesinkkey0, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+, columns.types=double}
              at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
              at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
      

      A more easier way to reproduce is

      set hive.cbo.enable=true;
      set hive.exec.check.crossproducts=false;
      
      set hive.stats.fetch.column.stats=true;
      set hive.auto.convert.join=false;
      
      select p_size, src.key
      from 
      part join src
      on p_size=key;
      

      As you can see, p_size is integer while src.key is string. Both of them should be cast to double when they join. When return path is off, this will happen before Join, at RS. However, when return path is on, this will be considered as an expression in Join. Thus, when reducer is collecting different types of keys from different join branches, it throws exception.

      Attachments

        1. HIVE-10455.03.patch
          9 kB
          Pengcheng Xiong
        2. HIVE-10455.02.patch
          10 kB
          Pengcheng Xiong
        3. HIVE-10455.01.patch
          6 kB
          Pengcheng Xiong

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            pxiong Pengcheng Xiong Assign to me
            pxiong Pengcheng Xiong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment