Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9434

Recommission a datanode with 500k blocks may pause NN for 30 seconds

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In BlockManager, processOverReplicatedBlocksOnReCommission is called within the namespace lock. There is a (not very useful) log message printed in processOverReplicatedBlock. When there is a large number of blocks stored in a storage, printing the log message for each block can pause NN to process any other operations. We did see that it could pause NN for 30 seconds for a storage with 500k blocks.

      I suggest to change the log message to trace level as a quick fix.

        Attachments

        1. h9434_20151116.patch
          1 kB
          Tsz-wo Sze
        2. h9434_20151116_branch-2.6.patch
          1 kB
          Tsz-wo Sze

          Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6425 Large postponedMisreplicatedBlocks has impact on blockReport latency

          • Major - Major loss of function.
          • Closed

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                szetszwo Tsz-wo Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: