Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13139

Default HDFS as Azure WASB tries rebalancing datanode data to HDFS (0% capacity) and fails

    XMLWordPrintableJSON

    Details

      Description

      ConfiguringĀ Azure WASB storage as a default HDFS location which means that Hadoop HDFS capacity will be 0. I have default replication as 1 but now when I am trying to decommission a node, datanode tries to rebalance some 28KB of data to another available datanode. However, our HDFS has 0 capacity and therefore, decommissioning fails with below given error:

      New node(s) could not be removed from the cluster. Reason Trying to move '28672' bytes worth of data to nodes with '0' bytes of capacity is not allowed

      Getting the information on cluster shows that default local HDFS is still used for some KB space which is getting rebalanced whereas available capacity is 0:

      "CapacityRemaining" : 0,
       "CapacityTotal" : 0,
       "CapacityUsed" : 131072,
       "DeadNodes" : "{}",
       "DecomNodes" : "{}",
       "HeapMemoryMax" : 1060372480,
       "HeapMemoryUsed" : 147668152,
       "NonDfsUsedSpace" : 0,
       "NonHeapMemoryMax" : -1,
       "NonHeapMemoryUsed" : 75319744,
       "PercentRemaining" : 0.0,
       "PercentUsed" : 100.0,
       "Safemode" : "",
       "StartTime" : 1518241019502,
       "TotalFiles" : 1,
       "UpgradeFinalized" : true,

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              abhi.sakhuja Abhishek Sakhuja
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: