Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4597

Under-replicated blocks are not calculated if the name-node is forced out of safe-mode.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.18.0
    • 0.18.3
    • None
    • None
    • Reviewed

    Description

      Currently during name-node startup under-replicated blocks are not added to the neededReplications queue until the name-node leaves safe mode. This is an optimization since otherwise all blocks will first go into the under-replicated queue and then most of them will be removed from it.
      When the name-node leaves safe-mode automatically it checks all blocks to have a correct number of replicas (processMisReplicatedBlocks()).
      When the name-node leaves safe-mode manually it does not perform the checkup.
      In the latter case all under-replicated blocks remain not replicated forever because there is no alternative mechanism to trigger replications.
      The proposal is to call processMisReplicatedBlocks() any time the name-node leaves safe mode - automatically or manually.
      In addition to solving that problem this could be an alternative mechanism for refreshing neededReplications and excessReplicateMap sets.

      Attachments

        1. NeededRepl-18.patch
          0.5 kB
          Konstantin Shvachko
        2. NeededRepl.patch
          0.6 kB
          Konstantin Shvachko

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shv Konstantin Shvachko
            shv Konstantin Shvachko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment