Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12175

Upgrade Kryo version to 3.0.x

    XMLWordPrintableJSON

Details

    Description

      Current version of kryo (2.22) has some issue (refer exception below and in HIVE-12174) with serializing ArrayLists generated using Arrays.asList(). We need to either replace all occurrences of Arrays.asList() or change the current StdInstantiatorStrategy. This issue is fixed in later versions and kryo community recommends using DefaultInstantiatorStrategy with fallback to StdInstantiatorStrategy. More discussion about this issue is here https://github.com/EsotericSoftware/kryo/issues/216. Alternatively, custom serilization/deserilization class can be provided for Arrays.asList.

      Also, kryo 3.0 introduced unsafe based serialization which claims to have much better performance for certain types of serialization.

      Exception:

      Caused by: java.lang.NullPointerException
      	at java.util.Arrays$ArrayList.size(Arrays.java:2847)
      	at java.util.AbstractList.add(AbstractList.java:108)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	... 57 more
      

      Attachments

        1. HIVE-12175.1.patch
          8 kB
          Prasanth Jayachandran
        2. HIVE-12175.2.patch
          10 kB
          Prasanth Jayachandran
        3. HIVE-12175.3.patch
          11 kB
          Prasanth Jayachandran
        4. HIVE-12175.3.patch
          10 kB
          Prasanth Jayachandran
        5. HIVE-12175.4.patch
          13 kB
          Prasanth Jayachandran
        6. HIVE-12175.5.patch
          20 kB
          Prasanth Jayachandran
        7. HIVE-12175.6.patch
          20 kB
          Prasanth Jayachandran

        Issue Links

          Activity

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: