[ZOOKEEPER-2247] Zookeeper service becomes unavailable when leader fails to write transaction log

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.4.9, 3.5.3, 3.6.0
• Labels: pull-request-available
• Assignee: Rakesh Radhakrishnan
• Reporter: Mohammad Arshad
• Created: 14/Aug/15 13:13
• Updated: 30/Jan/19 14:31
• Resolved: 13/Aug/16 13:59

Description

Zookeeper service becomes unavailable when leader fails to write transaction log. Bellow are the exceptions

2015-08-14 15:41:18,556 [myid:100] - ERROR [SyncThread:100:ZooKeeperCriticalThread@48] - Severe unrecoverable error, from thread : SyncThread:100
java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
	at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376)
	at org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:331)
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:380)
	at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:563)
	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:178)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer$ZooKeeperServerListenerImpl@500] - Thread SyncThread:100 exits, error code 1
2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer@523] - shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:SessionTrackerImpl@232] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:LeaderRequestProcessor@77] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:PrepRequestProcessor@1035] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:ProposalRequestProcessor@88] - Shutting down
2015-08-14 15:41:18,561 [myid:100] - INFO  [SyncThread:100:CommitProcessor@356] - Shutting down
2015-08-14 15:41:18,561 [myid:100] - INFO  [CommitProcessor:100:CommitProcessor@191] - CommitProcessor exited loop!
2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:Leader$ToBeAppliedRequestProcessor@915] - Shutting down
2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:FinalRequestProcessor@646] - shutdown of request processor complete
2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:SyncRequestProcessor@191] - Shutting down
2015-08-14 15:41:18,563 [myid:100] - INFO  [ProcessThread(sid:100 cport:-1)::PrepRequestProcessor@159] - PrepRequestProcessor exited loop!

After this exception Leader server still remains leader. After this non recoverable exception the leader should go down and let other followers become leader.

Issue Links

• relates to: ZOOKEEPER-2529 Flaky Test: org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader
• relates to: ZOOKEEPER-2452 Back-port ZOOKEEPER-1460 to 3.4 for IPv6 literal address support.
• relates to: ZOOKEEPER-1907 Improve Thread handling
• links to: GitHub Pull Request #65
• links to: GitHub Pull Request #65