Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3582

Kryo errors when using custom serialization

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: storm-core
    • Labels:
      None

      Description

      (this has been reported on the user list in January and previously by a different user)
       
      ------------------------------------------------------------------------------------
       
      My code works fine with Storm 1.x but the workers crash constantly with Storm 2.x 
       
      Some exceptions look like 
       
      com.esotericsoftware.kryo.KryoException: Buffer underflow.
      at com.esotericsoftware.kryo.io.Input.require(Input.java:199) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.io.Input.readString(Input.java:483) ~[kryo-3.0.3.jar:?]

      whereas others are
       
      2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the request
      {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 95
      at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
      at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
      at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-client-2.1.0.jar:2.1.0]_}}

      the class ID values are random integers. 

      {{}}
      The tuples in my topologies contain mostly standard classes like String; the only exception is the following class 
       
      https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41
       
      for which we specify a custom serialization for Kryo.
       
      My configurations contain
       

       topology.kryo.register:
      • com.digitalpebble.stormcrawler.Metadata

      to register the custom class.
       
      The user who had reported the problem first also used custom serialization.
       
       
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jnioche Julien Nioche
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: