[HIVE-16611] Kryo remove field is not working

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.1
• Component/s: Serializers/Deserializers
• Target Version/s: 3.0.0
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 09/May/17 00:02
• Updated: 09/May/17 00:07

Description

I have seen some instances where removeField in SerializationUtilities is not working

      removeField(kryo, Operator.class, "colExprMap");
      removeField(kryo, AbstractOperatorDesc.class, "statistics");

These 2 fields are not expected to be serialized but some stacktraces where custom UDFs are involved these fields are observed in the stacktrace.

genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) 
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) 
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) 
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
...(snip)...
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) 
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173) 
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062) 
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076) 

This needs further investigation.

Issue Links

• is related to: HIVE-15218 Kyro Exception on subsequent run of a query in LLAP mode