[SAMZA-1301] Investigate ZKClient exceptions in ZkJobCoordinator.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 0.15.0
• Assignee: Unassigned
• Reporter: Shanthoosh Venkataraman
• Created: 19/May/17 22:44
• Updated: 04/Jan/18 20:36

Description

ProcessorChange event handler in ZkJobCoordinator fails with the following exception. Halts the generation of JobModel in some executions. One possible sequence to trigger this.

LocalApplicationRunner runner = new LocalApplicationRunner(config);
runner.run(streamApp1);
runner.kill(streamApp1);
runner = new LocalApplicationRunner(config);
runner.run(streamApp1);

java.lang.IllegalStateException: ZkClient already closed!
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:676)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:672)
at org.apache.samza.zk.ZkUtils.getSortedActiveProcessorsZnodes(ZkUtils.java:132)
at org.apache.samza.zk.ZkUtils.getSortedActiveProcessorsIDs(ZkUtils.java:160)
at org.apache.samza.zk.ZkJobCoordinator.generateNewJobModel(ZkJobCoordinator.java:192)
at org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:125)
at org.apache.samza.zk.ZkJobCoordinator$LeaderElectorListenerImpl.lambda$onBecomingLeader$18(ZkJobCoordinator.java:239)
at org.apache.samza.zk.ZkJobCoordinator$LeaderElectorListenerImpl$$Lambda$53/731465117.run(Unknown Source)
at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$scheduleAfterDebounceTime$0(ScheduleAfterDebounceTime.java:89)
at org.apache.samza.zk.ScheduleAfterDebounceTime$$Lambda$54/1552634391.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
8056 [debounce-thread-0] ERROR org.apache.samza.zk.ZkJobCoordinator - Received exception from in JobCoordinator Processing!
java.lang.IllegalStateException: ZkClient already closed!
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:676)
at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:672)
at org.apache.samza.zk.ZkUtils.getSortedActiveProcessorsZnodes(ZkUtils.java:132)
at org.apache.samza.zk.ZkUtils.getSortedActiveProcessorsIDs(ZkUtils.java:160)
at org.apache.samza.zk.ZkJobCoordinator.generateNewJobModel(ZkJobCoordinator.java:192)
at org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:125)
at org.apache.samza.zk.ZkJobCoordinator$LeaderElectorListenerImpl.lambda$onBecomingLeader$18(ZkJobCoordinator.java:239)
at org.apache.samza.zk.ZkJobCoordinator$LeaderElectorListenerImpl$$Lambda$53/731465117.run(Unknown Source)
at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$scheduleAfterDebounceTime$0(ScheduleAfterDebounceTime.java:89)
at org.apache.samza.zk.ScheduleAfterDebounceTime$$Lambda$54/1552634391.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• is related to: SAMZA-1064 Standalone Samza with Zookeeper for Coordination