[HDFS-3894] QJM: testRecoverAfterDoubleFailures can be flaky due to IPC client caching

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: QuorumJournalManager (HDFS-3077)
• Fix Version/s: QuorumJournalManager (HDFS-3077)
• Component/s: test
• Target Version/s: QuorumJournalManager (HDFS-3077)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 06/Sep/12 06:47
• Updated: 13/Sep/12 23:00
• Resolved: 13/Sep/12 23:00

Description

TestQJMWithFaults.testRecoverAfterDoubleFailures fails really occasionally. Looking into it, the issue seems to be that it's possible by random chance for an IPC server port to be reused between two different iterations of the test loop. The client will then pick up and re-use the existing IPC connection to the old server. However, the old server was shut down and restarted, so the old IPC connection is stale (ie disconnected). This causes the new client to get an EOF when it sends the "format()" call.