[HIVE-12175] Upgrade Kryo version to 3.0.x

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Serializers/Deserializers
• Labels: TODOC2.0
• Assignee: Prasanth Jayachandran Assign to me
• Reporter: Prasanth Jayachandran
• Created: 14/Oct/15 13:24
• Updated: 16/Feb/16 23:52
• Resolved: 24/Nov/15 18:44

Description

Current version of kryo (2.22) has some issue (refer exception below and in HIVE-12174) with serializing ArrayLists generated using Arrays.asList(). We need to either replace all occurrences of Arrays.asList() or change the current StdInstantiatorStrategy. This issue is fixed in later versions and kryo community recommends using DefaultInstantiatorStrategy with fallback to StdInstantiatorStrategy. More discussion about this issue is here https://github.com/EsotericSoftware/kryo/issues/216. Alternatively, custom serilization/deserilization class can be provided for Arrays.asList.

Also, kryo 3.0 introduced unsafe based serialization which claims to have much better performance for certain types of serialization.

Exception:

Caused by: java.lang.NullPointerException
	at java.util.Arrays$ArrayList.size(Arrays.java:2847)
	at java.util.AbstractList.add(AbstractList.java:108)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	... 57 more

Issue Links

• Blocked: HIVE-11519 kryo.KryoException: Encountered unregistered
• is related to: HIVE-12174 Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
• is related to: HIVE-12551 Fix several kryo exceptions in branch-1
• is required by: HIVE-12302 Use KryoPool instead of thread-local caching
• relates to: SPARK-10793 Make spark's use/subclassing of hive more maintainable
• relates to: SPARK-11416 Upgrade kryo package to version 3.0

Sub-Tasks

• Make use of Kryo's Object-to-Object deep copy: Open