[HADOOP-11780] Prevent IPC reader thread death

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
• Component/s: ipc
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 31/Mar/15 19:33
• Updated: 01/Sep/17 00:42
• Resolved: 28/Sep/16 15:17

Description

Reader threads can die to a race condition with the responder thread. If the server's ipc handler cannot send a response in one write, it delegates sending the rest of the response to the responder thread.

The race occurs when the responder thread has an exception writing to the socket. The responder closes the socket. This wakes up the reader polling on the socket. If a CancelledKeyException is thrown, which is a runtime exception, the reader dies. All connections serviced by that reader are now in limbo until the client possibly times out. New connections play roulette as to whether they are assigned to a defunct reader.

Issue Links

• duplicates: HADOOP-13657 IPC Reader thread could silently die and leave NameNode unresponsive