Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11191

Datanode Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system.

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5.0
    • Fix Version/s: None
    • Component/s: hdfs
    • Environment:

      SLES 11SP3
      HDP 2.5.0

      Description

      In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system.

      hdfs@kimtest1:~> hdfs dfsadmin -report
      Configured Capacity: 239942369274 (223.46 GB)
      Present Capacity: 207894724602 (193.62 GB)
      DFS Remaining: 207894552570 (193.62 GB)
      DFS Used: 172032 (168 KB)
      DFS Used%: 0.00%
      Under replicated blocks: 0
      Blocks with corrupt replicas: 0
      Missing blocks: 0
      Missing blocks (with replication factor 1): 0

      -------------------------------------------------
      Live datanodes (3):

      Name: 172.26.79.87:50010 (kimtest3)
      Hostname: kimtest3
      Decommission Status : Normal
      Configured Capacity: 79980789758 (74.49 GB)
      DFS Used: 57344 (56 KB)
      Non DFS Used: 9528000512 (8.87 GB)
      DFS Remaining: 70452731902 (65.61 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 88.09%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 06:59:02 PST 2016

      Name: 172.26.80.38:50010 (kimtest4)
      Hostname: kimtest4
      Decommission Status : Normal
      Configured Capacity: 79980789758 (74.49 GB)
      DFS Used: 57344 (56 KB)
      Non DFS Used: 13010952192 (12.12 GB)
      DFS Remaining: 66969780222 (62.37 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 83.73%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 06:59:02 PST 2016

      Name: 172.26.79.86:50010 (kimtest2)
      Hostname: kimtest2
      Decommission Status : Normal
      Configured Capacity: 79980789758 (74.49 GB)
      DFS Used: 57344 (56 KB)
      Non DFS Used: 9508691968 (8.86 GB)
      DFS Remaining: 70472040446 (65.63 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 88.11%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 06:59:02 PST 2016

      If you see my datanode root file system size its only 38GB

      kimtest3:~ # df -h /
      Filesystem Size Used Avail Use% Mounted on
      /dev/mapper/system-root 38G 2.6G 33G 8% /

      kimtest4:~ # df -h /
      Filesystem Size Used Avail Use% Mounted on
      /dev/mapper/system-root 38G 4.2G 32G 12% /

      kimtest2:~ # df -h /
      Filesystem Size Used Avail Use% Mounted on
      /dev/mapper/system-root 38G 2.6G 33G 8% /

      The below is from hdfs-site.xml file

      <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
      </property>

      I have removed the other directory grid1 and restarted datanode process.

      <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///grid/hadoop/hdfs/dn</value>
      </property>

      Now the size is reflecting correctly

      hdfs@kimtest1:/grid> hdfs dfsadmin -report
      Configured Capacity: 119971184637 (111.73 GB)
      Present Capacity: 103947243517 (96.81 GB)
      DFS Remaining: 103947157501 (96.81 GB)
      DFS Used: 86016 (84 KB)
      DFS Used%: 0.00%
      Under replicated blocks: 0
      Blocks with corrupt replicas: 0
      Missing blocks: 0
      Missing blocks (with replication factor 1): 0

      -------------------------------------------------
      Live datanodes (3):

      Name: 172.26.79.87:50010 (kimtest3)
      Hostname: kimtest3
      Decommission Status : Normal
      Configured Capacity: 39990394879 (37.24 GB)
      DFS Used: 28672 (28 KB)
      Non DFS Used: 4764057600 (4.44 GB)
      DFS Remaining: 35226308607 (32.81 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 88.09%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 07:34:02 PST 2016

      Name: 172.26.80.38:50010 (kimtest4)
      Hostname: kimtest4
      Decommission Status : Normal
      Configured Capacity: 39990394879 (37.24 GB)
      DFS Used: 28672 (28 KB)
      Non DFS Used: 6505525248 (6.06 GB)
      DFS Remaining: 33484840959 (31.19 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 83.73%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 07:34:01 PST 2016

      Name: 172.26.79.86:50010 (kimtest2)
      Hostname: kimtest2
      Decommission Status : Normal
      Configured Capacity: 39990394879 (37.24 GB)
      DFS Used: 28672 (28 KB)
      Non DFS Used: 4754358272 (4.43 GB)
      DFS Remaining: 35236007935 (32.82 GB)
      DFS Used%: 0.00%
      DFS Remaining%: 88.11%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 2
      Last contact: Tue Nov 29 07:34:02 PST 2016

        Attachments

        1. HDFS-11191.01.patch
          22 kB
          Weiwei Yang
        2. HDFS-11191.02.patch
          31 kB
          Weiwei Yang
        3. HDFS-11191.03.patch
          25 kB
          Weiwei Yang
        4. HDFS-11191.04.patch
          26 kB
          Weiwei Yang
        5. HDFS-11191.05.patch
          26 kB
          Weiwei Yang
        6. HDFS-11191.06.patch
          26 kB
          Weiwei Yang
        7. HDFS-11191.07.patch
          27 kB
          Weiwei Yang

          Issue Links

            Activity

              People

              • Assignee:
                cheersyang Weiwei Yang
                Reporter:
                dchander@hortonworks.com Deepak Chander
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: