Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16611

Kryo remove field is not working

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.1
    • Fix Version/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      I have seen some instances where removeField in SerializationUtilities is not working

            removeField(kryo, Operator.class, "colExprMap");
            removeField(kryo, AbstractOperatorDesc.class, "statistics");
      

      These 2 fields are not expected to be serialized but some stacktraces where custom UDFs are involved these fields are observed in the stacktrace.

      genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
      chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
      colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) 
      childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) 
      aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) 
      at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
      ...(snip)...
      at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) 
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173) 
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062) 
      at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076) 
      

      This needs further investigation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                prasanth_j Prasanth Jayachandran
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: