Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6945

BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.8.0
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I'm seeing ExcessBlocks metric increases to more than 300K in some clusters, however, there are no over-replicated blocks (confirmed by fsck).

      After a further research, I noticed when deleting a block, BlockManager does not remove the block from excessReplicateMap or decrement excessBlocksCount.
      Usually the metric is decremented when processing block report, however, if the block has been deleted, BlockManager does not remove the block from excessReplicateMap or decrement the metric.
      That way the metric and excessReplicateMap can increase infinitely (i.e. memory leak can occur).

      1. HDFS-6945-005.patch
        3 kB
        Akira AJISAKA
      2. HDFS-6945-004.patch
        3 kB
        Akira AJISAKA
      3. HDFS-6945-003.patch
        3 kB
        Akira AJISAKA
      4. HDFS-6945.2.patch
        3 kB
        Akira AJISAKA
      5. HDFS-6945.patch
        3 kB
        Akira AJISAKA

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Akira AJISAKA
            Reporter:
            Akira AJISAKA
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development