Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13356

Balancer:Set default value of minBlockSize to 10mb

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

       It seems we can run into a problem while a rolling upgrade with this.
      The Balancer is upgraded after NameNodes, so once NN is upgraded it will expect minBlockSize parameter in getBlocks(). The Balancer cannot send it yet, so NN will use the default, which you set to 0. So NN will start unexpectedly sending small blocks to the Balancer. So we should

      1. either change the default in protobuf to 10 MB
      2. or treat minBlockSize == 0 in NameNodeRpcServer as a signal to use the configuration variable DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY.

      If you agree, we should create a follow up jira. I wanted to backport this down the chain of branches, but this upgrade scenario is stopping me.

      Konstantin Shvachko commented this in  HDFS-13222 jira.

      https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855

      Attachments

        1. HDFS-13356.02.patch
          0.9 kB
          Bharat Viswanadham
        2. HDFS-13356.01.patch
          0.9 kB
          Bharat Viswanadham
        3. HDFS-13356.00.patch
          0.9 kB
          Bharat Viswanadham

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bharat Bharat Viswanadham
            bharat Bharat Viswanadham
            Votes:
            0 Vote for this issue
            Watchers:
            7 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment