Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7711

Error Serializing GenericUDF

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.13.0
    • None
    • None
    • None

    Description

      I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was ok in HIVE 0.12.0).

      The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized using Kryo, trying to serialize stuff in my GenericUDF which is not serializable (doesn't implement Serializable).

      Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte:
      "/**

      • In case genericUDF is Serializable, we will serialize the object.
        *
      • In case genericUDF does not implement Serializable, Java will remember the
      • class of genericUDF and creates a new instance when deserialized. This is
      • exactly what we want.
        */"

      Find the stacktrace below, however, the description above should be clear.

      Exception in thread "main" org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
      Serialization trace:
      value (java.util.concurrent.atomic.AtomicReference)
      state (clojure.lang.Atom)
      state (udfs.ArraySum)
      genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
      aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
      mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826)
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840)
      at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:738)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.UnsupportedOperationException
      at clojure.lang.APersistentMap.put(APersistentMap.java:287)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:140)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      ... 41 more

      Attachments

        1. HIVE-7711.1.patch.txt
          3 kB
          Navis Ryu

        Activity

          People

            Unassigned Unassigned
            cbbetz Dr. Christian Betz
            Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: