Uploaded image for project: 'Hivemall'
  1. Hivemall
  2. HIVEMALL-236

to_json cause KryoException/NullPointerException with ArrayList due to Kryo bug

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.2
    • Fix Version/s: 0.6.0
    • Labels:
      None

      Description

      SELECT
        from_json(
          '{ "person" : { "name" : "makoto" , "age" : 37 } }',
          'struct<name:string,age:int>', 
          array('person')
        )

      to_json cause the following Kryo exception depending on underlying Kryo version.

      Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
      Serialization trace:
      columnNames (hivemall.tools.json.ToJsonUDF)
      genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
      aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:915)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:823)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:837)
      at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:330)
      ... 17 more
      Caused by: java.lang.NullPointerException
      at java.util.Arrays$ArrayList.size(Arrays.java:3818)
      at java.util.AbstractList.add(AbstractList.java:108)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:105)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      ... 44 more

      BTW, the following query will run without error:

      select
        from_json(
          '[0.1,1.1,2.2]',
          'array<double>'
        ),

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                myui Makoto Yui
                Reporter:
                myui Makoto Yui
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: