Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10309

Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: balancer & mover
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      While running HDFS Balancer I get error given below when dfs.blockSize is defined with suffix k(kilo), m(mega), g(giga) in hdfs-site.xml. In my deployment dfs.blocksize is set to 128m.

      hdfs@bcpc-vm1:/home/ubuntu$ hdfs balancer
      16/04/19 08:49:51 INFO balancer.Balancer: namenodes  = [hdfs://Test-Laptop]
      16/04/19 08:49:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source 
      nodes = 0, #blockpools = 0, run during upgrade = false]
      16/04/19 08:49:51 INFO balancer.Balancer: included nodes = []
      16/04/19 08:49:51 INFO balancer.Balancer: excluded nodes = []
      16/04/19 08:49:51 INFO balancer.Balancer: source nodes = []
      Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
      16/04/19 08:49:52 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
      16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
      16/04/19 08:49:52 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
      16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
      16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
      Apr 19, 2016 8:49:52 AM  Balancing took 1.408 seconds
      16/04/19 08:49:52 ERROR balancer.Balancer: Exiting balancer due an exception
      java.lang.NumberFormatException: For input string: "128m"
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
              at java.lang.Long.parseLong(Long.java:589)
              at java.lang.Long.parseLong(Long.java:631)
              at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1311)
              at org.apache.hadoop.hdfs.server.balancer.Balancer.getLong(Balancer.java:221)
              at org.apache.hadoop.hdfs.server.balancer.Balancer.<init>(Balancer.java:281)
              at org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:660)
              at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:774)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:903)
      

      However, the workaround for this is to run hdfs balancer with passing numeric value for dfs.blocksize or change your hdfs-site.xml.

      hdfs balancer -Ddfs.blocksize=134217728
      

        Attachments

        1. HDFS-10309.01.patch
          1 kB
          Amit Anand
        2. HDFS-10309.02.patch
          2 kB
          Amit Anand

          Activity

            People

            • Assignee:
              aanand001c Amit Anand
              Reporter:
              aanand001c Amit Anand
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: