[METRON-1551] Profiler Should Not Use Java Serialization

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.5.0
• Assignee: Nick Allen
• Reporter: Nick Allen
• Created: 11/May/18 19:21
• Updated: 28/Aug/18 20:01
• Resolved: 22/May/18 19:23

Description

When running the Profiler in a topology where serialization occurs, the following error happens.  This can occur when the number of workers is greater than 1.

The topology should not be using Java serialization for serializing tuple values as this will negatively impact performance. 

2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult
 at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult
 at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?]
 at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 ... 6 more
Caused by: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[?:1.8.0_162]
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) ~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) ~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) ~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_162]
 at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?]
 at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 ... 6 more

 

 

Issue Links

• is related to: METRON-1757 Storm Profiler Serialization Exception
• links to: GitHub Pull Request #1012