Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13356

Balancer:Set default value of minBlockSize to 10mb

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

       It seems we can run into a problem while a rolling upgrade with this.
      The Balancer is upgraded after NameNodes, so once NN is upgraded it will expect minBlockSize parameter in getBlocks(). The Balancer cannot send it yet, so NN will use the default, which you set to 0. So NN will start unexpectedly sending small blocks to the Balancer. So we should

      1. either change the default in protobuf to 10 MB
      2. or treat minBlockSize == 0 in NameNodeRpcServer as a signal to use the configuration variable DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY.

      If you agree, we should create a follow up jira. I wanted to backport this down the chain of branches, but this upgrade scenario is stopping me.

      Konstantin Shvachko commented this in  HDFS-13222 jira.

      https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855

        Attachments

        1. HDFS-13356.02.patch
          0.9 kB
          Bharat Viswanadham
        2. HDFS-13356.01.patch
          0.9 kB
          Bharat Viswanadham
        3. HDFS-13356.00.patch
          0.9 kB
          Bharat Viswanadham

        Issue Links

          Activity

            People

            • Assignee:
              bharat Bharat Viswanadham
              Reporter:
              bharat Bharat Viswanadham

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment