Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8825 Enhancements to Balancer
  3. HDFS-8875

Optimize the wait time in Balancer for federation scenario

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Balancer has wait time between two consecutive iterations. That is to give some time for block movement to be fully committed ( return from replaceBlock doesn't mean the NN's blockmap has been updated and the block has been invalidated on the source node.).

      This wait time could be 23 seconds if dfs.heartbeat.interval is set to 10 and dfs.namenode.replication.interval is to 3. In the case of federation, given we iterate through all namespaces in each iteration, this wait time becomes unnecessary as while balancer is processing the next namespace, it gives the previous namespace it just finished time to commit.

      In addition, Balancer calls Collections.shuffle(connectors); It doesn't seem necessary.

        Attachments

          Activity

            People

            • Assignee:
              ctrezzo Chris Trezzo
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: