Hive
  1. Hive
  2. HIVE-6412

SMB join on Decimal columns causes cast exception in JoinUtil.computeKeys

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
              at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:49)
              at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:27)
              at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:281)
              at org.apache.hadoop.hive.ql.exec.JoinUtil.computeKeys(JoinUtil.java:143)
              at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.next(SMBMapJoinOperator.java:809)
              at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.nextHive(SMBMapJoinOperator.java:771)
              at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.setupContext(SMBMapJoinOperator.java:710)
              at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.setUpFetchContexts(SMBMapJoinOperator.java:538)
              at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:248)
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
              at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
              at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
      

      Repro:

      create table vsmb_bucket_1(key decimal(9,0), value decimal(38,10)) 
        CLUSTERED BY (key) 
        SORTED BY (key) INTO 1 BUCKETS 
        STORED AS ORC;
      create table vsmb_bucket_2(key decimal(19,3), value decimal(28,0)) 
        CLUSTERED BY (key) 
        SORTED BY (key) INTO 1 BUCKETS 
        STORED AS ORC;
        
      insert into table vsmb_bucket_1 
        select cast(cint as decimal(9,0)) as key, 
          cast(cfloat as decimal(38,10)) as value 
        from alltypesorc limit 2;
      insert into table vsmb_bucket_2 
        select cast(cint as decimal(19,3)) as key, 
          cast(cfloat as decimal(28,0)) as value 
        from alltypesorc limit 2;
      
      set hive.optimize.bucketmapjoin = true;
      set hive.optimize.bucketmapjoin.sortedmerge = true;
      set hive.auto.convert.sortmerge.join.noconditionaltask = true;
      set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
      
      explain
      select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
      select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
      

        Activity

        Xuefu Zhang made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Cannot Reproduce [ 5 ]
        Xuefu Zhang made changes -
        Assignee Xuefu Zhang [ xuefuz ]
        Xuefu Zhang made changes -
        Description Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
                at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:49)
                at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:27)
                at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:281)
                at org.apache.hadoop.hive.ql.exec.JoinUtil.computeKeys(JoinUtil.java:143)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.next(SMBMapJoinOperator.java:809)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.nextHive(SMBMapJoinOperator.java:771)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.setupContext(SMBMapJoinOperator.java:710)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.setUpFetchContexts(SMBMapJoinOperator.java:538)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:248)
                at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
                at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
                at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
                at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)


        Repro:
        create table vsmb_bucket_1(key decimal(9,0), value decimal(38,10))
          CLUSTERED BY (key)
          SORTED BY (key) INTO 1 BUCKETS
          STORED AS ORC;
        create table vsmb_bucket_2(key decimal(19,3), value decimal(28,0))
          CLUSTERED BY (key)
          SORTED BY (key) INTO 1 BUCKETS
          STORED AS ORC;
          
        insert into table vsmb_bucket_1
          select cast(cint as decimal(9,0)) as key,
            cast(cfloat as decimal(38,10)) as value
          from alltypesorc limit 2;
        insert into table vsmb_bucket_2
          select cast(cint as decimal(19,3)) as key,
            cast(cfloat as decimal(28,0)) as value
          from alltypesorc limit 2;

        set hive.optimize.bucketmapjoin = true;
        set hive.optimize.bucketmapjoin.sortedmerge = true;
        set hive.auto.convert.sortmerge.join.noconditionaltask = true;
        set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;

        explain
        select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
        select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
        {code}
        Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
                at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:49)
                at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveWritableObject(JavaHiveDecimalObjectInspector.java:27)
                at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:281)
                at org.apache.hadoop.hive.ql.exec.JoinUtil.computeKeys(JoinUtil.java:143)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.next(SMBMapJoinOperator.java:809)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.nextHive(SMBMapJoinOperator.java:771)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator$MergeQueue.setupContext(SMBMapJoinOperator.java:710)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.setUpFetchContexts(SMBMapJoinOperator.java:538)
                at org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:248)
                at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
                at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
                at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
                at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
        {code}

        Repro:
        {code}
        create table vsmb_bucket_1(key decimal(9,0), value decimal(38,10))
          CLUSTERED BY (key)
          SORTED BY (key) INTO 1 BUCKETS
          STORED AS ORC;
        create table vsmb_bucket_2(key decimal(19,3), value decimal(28,0))
          CLUSTERED BY (key)
          SORTED BY (key) INTO 1 BUCKETS
          STORED AS ORC;
          
        insert into table vsmb_bucket_1
          select cast(cint as decimal(9,0)) as key,
            cast(cfloat as decimal(38,10)) as value
          from alltypesorc limit 2;
        insert into table vsmb_bucket_2
          select cast(cint as decimal(19,3)) as key,
            cast(cfloat as decimal(28,0)) as value
          from alltypesorc limit 2;

        set hive.optimize.bucketmapjoin = true;
        set hive.optimize.bucketmapjoin.sortedmerge = true;
        set hive.auto.convert.sortmerge.join.noconditionaltask = true;
        set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;

        explain
        select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
        select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key;
        {code}
        Remus Rusanu made changes -
        Field Original Value New Value
        Summary SMB join on Decimal columns causes caste exception in JoinUtil.computeKeys SMB join on Decimal columns causes cast exception in JoinUtil.computeKeys
        Remus Rusanu created issue -

          People

          • Assignee:
            Xuefu Zhang
            Reporter:
            Remus Rusanu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development