Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15622

Deleted blocks linger in the replications queue

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.2, 3.3.1, 3.4.0, 3.2.3
    • Component/s: hdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We had incident whereas after resolving a missing blocks incident by restarting two dead nodes, there were still 8 missing, but the list was empty. Metasave shows the 8 blocks are "orphaned" meaning the files were already deleted. It is unclear why they were left in the replication queue.

      • The containing node was flaky and started stoped multiple time.
      • The block allocation didn't work well due to the cluster-level storage space exhaustion.
      • The NN was in safe mode.

      Triggering a full block report from the node didn't have any effect. It will clear up if a failover happens as the repl queue will be reinitialized.

        Attachments

        1. HDFS-15622.001.patch
          8 kB
          Ahmed Hussein
        2. HDFS-15622.002.patch
          8 kB
          Ahmed Hussein

          Activity

            People

            • Assignee:
              ahussein Ahmed Hussein
              Reporter:
              ahussein Ahmed Hussein
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: