[ZOOKEEPER-3598] Fix potential data inconsistency issue due to CommitProcessor not gracefully shutdown

• Type: Bug
• Status: Reopened
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available

Release Note

Committed to master branch as 79f99af81842f415b97e1c3c18c953df5bd129b2

(I had a problem with the script and JIRA)

• Assignee: Fangmin Lv
• Reporter: Fangmin Lv
• Created: 29/Oct/19 19:23
• Updated: 18/Nov/19 12:41

Description

 
There was a regression introduced after inline the write in CommitProcessor with changes intorduced in ZOOKEEPER-3359, which didn't wait the in-flight write to finish before calling shutdown on the nextProcessor.
 
So it's possible that CommitProcessor thread and QuorumPeer thread will update the DataTree concurrently if we're doing fastForwardDataBase at the end of ZooKeeperServer.shutdown, which will cause inconsistent issue.
 
This JIRA is going to make sure we wait on the CommitProcessor to shutdown gracefully before calling shutdown on next processor, and exit if we cannot finish it gracefully to avoid potential inconsistency.

Issue Links

• links to: GitHub Pull Request #1130
• links to: GitHub Pull Request #1143