Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15253

Set default throttle value on dfs.image.transfer.bandwidthPerSec

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Release Note:
      The configuration dfs.image.transfer.bandwidthPerSec which defines the maximum bandwidth available for fsimage transfer is changed from 0 (meaning no throttle at all) to 50MB/s.

      Description

      The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can use maximum available bandwidth for fsimage transfers during checkpoint. I think we should throttle this. Many users were experienced namenode failover when transferring large image size along with fsimage replication on dfs.namenode.name.dir. eg. >25Gb.  

      Thought to set,

      dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s)

      dfs.namenode.checkpoint.txns=2000000 (Default is 1M, good to avoid frequent checkpoint. However, the default checkpoint runs every 6 hours once)

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kpalanisamy Karthik Palanisamy
                Reporter:
                kpalanisamy Karthik Palanisamy
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m