Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1757

Storm Profiler Serialization Exception

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Labels:
      None

      Description

      When running the Storm Profiler with multiple workers this serialization error can occur.

      Even in an environment with multiple workers It seems to not occur all the time. It may depend on how the individual bolts are executed across the workers in a cluster. This likely occurs only when the HBaseBolt is executed on a separate worker from the ProfileBuilderBolt.

      It will never occur when running with a single worker.

      2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the request
      com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.metron.common.configuration.profiler.ProfileResult
      Serialization trace:
      result (org.apache.metron.common.configuration.profiler.ProfileConfig)
      definition (org.apache.metron.profiler.ProfileMeasurement)
       at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
       at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
       at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.messaging.netty.Server.received(Server.java:254) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nickwallen Nick Allen
                Reporter:
                nickwallen Nick Allen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: