Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6489

DFS Used space is not correct computed on frequent append operations

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0, 2.7.1, 2.7.2
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      The current implementation of the Datanode will increase the DFS used space on each block write operation. This is correct in most scenario (create new file), but sometimes it will behave in-correct(append small data to a large block).
      For example, I have a file with only one block(say, 60M). Then I try to append to it very frequently but each time I append only 10 bytes;
      Then on each append, dfs used will be increased with the length of the block(60M), not teh actual data length(10bytes).
      Consider in a scenario I use many clients to append concurrently to a large number of files (1000+), assume the block size is 32M (half of the default value), then the dfs used will be increased 1000*32M = 32G on each append to the files; but actually I only write 10K bytes; this will cause the datanode to report in-sufficient disk space on data write.

      2014-06-04 15:27:34,719 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Insufficient space for appending to FinalizedReplica, blk_1073742834_45306, FINALIZED

      But the actual disk usage:

      [root@hdsh143 ~]# df -h
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda3 16G 2.9G 13G 20% /
      tmpfs 1.9G 72K 1.9G 1% /dev/shm
      /dev/sda1 97M 32M 61M 35% /boot

        Attachments

        1. HDFS-6489.001.patch
          9 kB
          Weiwei Yang
        2. HDFS-6489.002.patch
          12 kB
          Weiwei Yang
        3. HDFS-6489.003.patch
          14 kB
          Weiwei Yang
        4. HDFS-6489.004.patch
          12 kB
          Weiwei Yang
        5. HDFS-6489.005.patch
          4 kB
          Ravi Prakash
        6. HDFS-6489.006.patch
          6 kB
          Ravi Prakash
        7. HDFS-6489.007.patch
          7 kB
          Ravi Prakash
        8. HDFS6489.java
          1 kB
          Bogdan Raducanu

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stanley_shi stanley shi
              • Votes:
                2 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: