[ZOOKEEPER-1109] Zookeeper service is down when SyncRequestProcessor meets any exception.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.3.1, 3.3.2, 3.3.3
• Fix Version/s: 3.4.0
• Component/s: quorum
• Hadoop Flags: Reviewed
• Tags: quorum, leader, disk full, shutdown
• Assignee: Laxman
• Reporter: Laxman
• Created: 24/Jun/11 04:48
• Updated: 23/Nov/11 19:22
• Resolved: 25/Jul/11 21:01

Description

Problem Zookeeper is not shut down completely when dataDir disk space is full and ZK Cluster went into unserviceable state.

Scenario
If the leader zookeeper disk is made full, the zookeeper is trying to shutdown. But it is waiting indefinitely while shutting down the SyncRequestProcessor thread.

Root Cause
this.join() is invoked in the same thread where System.exit(11) has been triggered.

When disk space full happens, It got the exception as follows 'No space left on device' and invoked System.exit(11) from the SyncRequestProcessor thread(The following logs shows the same). Before exiting JVM, ZK will execute the ShutdownHook of QuorumPeerMain and the flow comes to SyncRequestProcessor.shutdown(). Here this.join() is invoked in the same thread where System.exit(11) has been invoked.

Issue Links

• is related to: ZOOKEEPER-121 SyncRequestProcessor is not closing log stream during shutdown