[STORM-2979] WorkerHooks EOFException during run_worker_shutdown_hooks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.6
• Fix Version/s: 2.0.0, 1.1.3, 1.0.7, 1.2.2
• Component/s: storm-core
• Labels: pull-request-available
• Environment: centos 7 ; storm-core 1.0.6 ; eclipse Mars2 ; java 1.8.0_151
• Assignee: Michel hummel
• Reporter: Robin Perice
• Created: 27/Feb/18 12:28
• Updated: 11/Apr/18 19:09
• Resolved: 11/Apr/18 12:51

Description

Hi,

I'm trying to use the BaseWorkerHook but an exception is thrown after I killed the topology.

The issue is exactly the same as : http://user.storm.apache.narkive.com/uchOrwlH/workerhook-deserialization-problem

An extract of my code :

 

// topology
final TridentTopology topology = new TridentTopology();
// ... I skip all the topology configuration part
final StormTopology topo = topology.build();

// hook
final BaseWorkerHook hook = new BaseWorkerHook();
final ByteBuffer serializedHook = ByteBuffer.wrap(Utils.javaSerialize(hook ));
topo.add_to_worker_hooks(hook);

// submit topology
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(name,config,topo);
Utils.sleep(60000);

// kill topology
final KillOptions killOptions = new KillOptions();
killOptions.set_wait_secs(0);
cluster.killTopologyWithOpts(name, killOptions);
Utils.sleep(10000);
cluster.shutdown();

 

I have the following error :

java.lang.RuntimeException: java.io.EOFException
    at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:254)
    at org.apache.storm.daemon.worker$run_worker_shutdown_hooks$iter__5456__5460$fn__5461.invoke(worker.clj:578)
    at clojure.lang.LazySeq.sval(LazySeq.java:40)
    at clojure.lang.LazySeq.seq(LazySeq.java:49)
    at clojure.lang.RT.seq(RT.java:507)
    at clojure.core$seq__4128.invoke(core.clj:137)
    at clojure.core$dorun.invoke(core.clj:3009)
    at clojure.core$doall.invoke(core.clj:3025)
    at org.apache.storm.daemon.worker$run_worker_shutdown_hooks.invoke(worker.clj:576)
    at org.apache.storm.daemon.worker$fn__5471$exec_fn__1371__auto__$reify__5473$shutdown_STAR___5493.invoke(worker.clj:693)
    at org.apache.storm.daemon.worker$fn__5471$exec_fn__1371__auto__$reify$reify__5519.shutdown(worker.clj:706)
    at org.apache.storm.ProcessSimulator.killProcess(ProcessSimulator.java:67)
    at org.apache.storm.daemon.supervisor.LocalContainer.kill(LocalContainer.java:59)
    at org.apache.storm.daemon.supervisor.Slot.killContainerForChangedAssignment(Slot.java:311)
    at org.apache.storm.daemon.supervisor.Slot.handleRunning(Slot.java:527)
    at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:265)
    at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:741)
Caused by: java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
    at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:245)
    ... 16 more

 

Maybe it is related to log4j shutdown hooks (https://issues.apache.org/jira/browse/STORM-2176) so I tried to disable the hook in my src/test/resources/log4j2.xml.

 

<Configuration monitorInterval="60" shutdownHook="disable">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        </Console>
    </Appenders>

    <Loggers>
        <Root level="debug">
            <AppenderRef ref="Console" />
        </Root>
    </Loggers>
</Configuration>

But it does not change anything.

 

Of course the purpose of my work is to use my own worker hook extending the BaseWorkerHook.

 

 

 

 

Issue Links

• links to: GitHub Pull Request #2591
• links to: GitHub Pull Request #2625