[SAMZA-1364] Unclean shutdown of streamApplication in LocalApplicationRunner.kill(streamApplication)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.1
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 19/Jul/17 20:20
• Updated: 26/Jul/17 00:29
• Resolved: 26/Jul/17 00:29

Description

In some cases LocalAppRunner.waitForFinish indefinitely blocks after LocalApplicationRunner.kill. Last step in LocalAppRunner.kill(streamApp) is zkClient.close()[zkClient belongs to ZkCoordinationService].

ApplicationRunner.kill triggers listeners chain & in final listener zkclient.close throws ZkInterruptedException (RuntimeException) & it's swallowed in listeners preventing shutdownLatch update in LocalApplicationRunner(required for proper shutdown).


Exception log:


org.I0Itec.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException
    at org.I0Itec.zkclient.ZkClient.close(ZkClient.java:1277)
    at org.apache.samza.zk.ZkUtils.close(ZkUtils.java:296)
    at org.apache.samza.zk.ZkCoordinationUtils.reset(ZkCoordinationUtils.java:47)