Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8950

NameNode refresh doesn't remove DataNodes that are no longer in the allowed list

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission.

      To reproduce:
      1. Add a DN to dfs_hosts_allow
      2. Refresh NN
      3. Start DN. Now NN starts seeing DN.
      4. Stop DN
      5. Remove DN from dfs_hosts_allow
      6. Refresh NN -> NN is still reporting DN as being used by HDFS.

      This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly.

      Attachments

        1. HDFS-8950.001.patch
          8 kB
          Daniel Templeton
        2. HDFS-8950.002.patch
          8 kB
          Daniel Templeton
        3. HDFS-8950.003.patch
          10 kB
          Daniel Templeton
        4. HDFS-8950.004.patch
          10 kB
          Daniel Templeton
        5. HDFS-8950.005.patch
          12 kB
          Daniel Templeton
        6. HDFS-8950.branch-2.7.patch
          12 kB
          Kihwal Lee

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            templedf Daniel Templeton Assign to me
            templedf Daniel Templeton
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment