Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7172

TestSplitLogManager.testVanishingTaskZNode() fails when run individually and is flaky

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.4, 0.95.2
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestSplitLogManager.testVanishingTaskZNode fails when run individually (run just that test case from eclipse). I've also noticed that it is flaky on windows.

      The reason is a rare race condition, which somehow does not happen that much when the whole class is run.

      The sequence of events is smt like this:

      • we create 1 log file to split
      • we call splitLogDistributed() in its own thread.
      • splitLogDistributed() is waiting in waitForSplittingCompletion() since there are no splitlogworkers, it keep waiting.
      • we delete the task znode from zk
      • SplitLogManager receives the zk callback from GetDataAsyncCallback, which will call setDone() and mark the task as success.
      • However, meanwhile the waitForSplittingCompletion() loops sees that remainingInZK == 0, and calls return concurrently to the above.
      • on return from waitForSplittingCompletion(), splitLogDistributed() fails because the znode delete callback has not completed yet.

      This race only happens when the last task is deleted from zk, and normally only the SplitLogManager deletes the task znodes after processing it, so I don't think this is a production issue.

        Attachments

        1. hbase-7172_v1.patch
          0.8 kB
          Enis Soztutar
        2. hbase-7172_v2.patch
          12 kB
          Enis Soztutar
        3. hbase-7172_v2-0.94.patch
          9 kB
          Enis Soztutar

        Issue Links

          Activity

            People

            • Assignee:
              enis Enis Soztutar
              Reporter:
              enis Enis Soztutar

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment