Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15622

Deleted blocks linger in the replications queue

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.2, 3.3.1, 3.4.0, 3.2.3
    • hdfs
    • None
    • Reviewed

    Description

      We had incident whereas after resolving a missing blocks incident by restarting two dead nodes, there were still 8 missing, but the list was empty. Metasave shows the 8 blocks are "orphaned" meaning the files were already deleted. It is unclear why they were left in the replication queue.

      • The containing node was flaky and started stoped multiple time.
      • The block allocation didn't work well due to the cluster-level storage space exhaustion.
      • The NN was in safe mode.

      Triggering a full block report from the node didn't have any effect. It will clear up if a failover happens as the repl queue will be reinitialized.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ahussein Ahmed Hussein Assign to me
            ahussein Ahmed Hussein
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment