[HBASE-7172] TestSplitLogManager.testVanishingTaskZNode() fails when run individually and is flaky

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.94.4, 0.95.2
• Fix Version/s: 0.94.4, 0.95.0
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 16/Nov/12 00:38
• Updated: 26/Feb/13 08:23
• Resolved: 28/Nov/12 22:41

Description

TestSplitLogManager.testVanishingTaskZNode fails when run individually (run just that test case from eclipse). I've also noticed that it is flaky on windows.

The reason is a rare race condition, which somehow does not happen that much when the whole class is run.

The sequence of events is smt like this:

• we create 1 log file to split
• we call splitLogDistributed() in its own thread.
• splitLogDistributed() is waiting in waitForSplittingCompletion() since there are no splitlogworkers, it keep waiting.
• we delete the task znode from zk
• SplitLogManager receives the zk callback from GetDataAsyncCallback, which will call setDone() and mark the task as success.
• However, meanwhile the waitForSplittingCompletion() loops sees that remainingInZK == 0, and calls return concurrently to the above.
• on return from waitForSplittingCompletion(), splitLogDistributed() fails because the znode delete callback has not completed yet.

This race only happens when the last task is deleted from zk, and normally only the SplitLogManager deletes the task znodes after processing it, so I don't think this is a production issue.

Issue Links

• is related to: HBASE-7485 TestSplitLogManager is still flaky on windows
• relates to: HBASE-7165 TestSplitLogManager.testUnassignedTimeout is flaky