Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13677

org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException when folding CASE expression

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      When folding CASE expressions, children for new expressions are created using the sublist method in List class. Subsequently this leads to problems in Serialization as both list and sublist are modified. In particular, the following Exception is thrown:

      org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
      Serialization trace:
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator)
      reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
      org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
      Serialization trace:
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator)
      reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:445)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:319)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor$1.call(ReduceRecordProcessor.java:98)
      	at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.<init>(ReduceRecordProcessor.java:95)
      	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:136)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
      	at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1230)
      	at java.util.ArrayList$SubList.size(ArrayList.java:1040)
      	at java.util.AbstractList.add(AbstractList.java:108)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	... 51 more
      

      To reproduce, the following can be done:

      drop table test1; 
      drop table test2;
      
      create table test1 (id int, desc string) stored as orc; 
      create table test2 (id int, desc string) stored as orc;
      
      select 
      case 
      when (case when a.desc='test' then 1 else 0 end)=0 then 'test' 
      else null 
      end as val 
      FROM test1 a 
      JOIN test2 b ON a.id=b.id;
      

        Attachments

        1. HIVE-13677.patch
          6 kB
          Jesus Camacho Rodriguez

          Activity

            People

            • Assignee:
              jcamachorodriguez Jesus Camacho Rodriguez
              Reporter:
              jcamachorodriguez Jesus Camacho Rodriguez
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: