[ZOOKEEPER-1907] Improve Thread handling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.4.7, 3.5.1, 3.6.0
• Component/s: server
• Assignee: Rakesh Radhakrishnan
• Reporter: Rakesh Radhakrishnan
• Created: 31/Mar/14 15:18
• Updated: 07/Dec/16 03:03
• Resolved: 17/Dec/15 09:13

Description

Server has many critical threads running and co-ordinating each other like RequestProcessor chains et. When going through each threads, most of them having the similar structure like:

public void run() {
        try {
              while(running)
                   // processing logic
              }
        } catch (InterruptedException e) {
            LOG.error("Unexpected interruption", e);
        } catch (Exception e) {
            LOG.error("Unexpected exception", e);
        }
        LOG.info("...exited loop!");
}

From the design I could see, there could be a chance of silently leaving the thread by swallowing the exception. If this happens in the production, the server would get hanged forever and would not be able to deliver its role. Now its hard for the management tool to detect this.

The idea of this JIRA is to discuss and imprv.

Reference: Community discussion thread

Issue Links

• is related to: ZOOKEEPER-602 log all exceptions not caught by ZK threads
• is related to: ZOOKEEPER-2247 Zookeeper service becomes unavailable when leader fails to write transaction log
• is related to: ZOOKEEPER-207 All the threads should have names so that its easier to see throguh a stack trace.
• relates to: ZOOKEEPER-2029 Leader.LearnerCnxAcceptor should handle exceptions in run()
• links to: GitHub Pull Request #39