Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3061

Backport HDFS-1487 to branch-1

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.20.203.0, 0.23.0, 1.0.2
    • 1.0.3
    • namenode
    • None

    Description

      It appears that there's a condition under which a HDFS directory with a space quota set can get to a point where the cached size for the directory can permanently differ from the computed value. When this happens the following command:

      hadoop fs -count -q /tmp/quota-test
      

      results in the following output in the NameNode logs:

      WARN org.apache.hadoop.hdfs.server.namenode.NameNode: Inconsistent diskspace for directory quota-test. Cached: 6000 Computed: 6072
      

      I've observed both transient and persistent instances of this happening. In the transient instances this warning goes away, but in the persistent instances every invocation of the fs -count -q command yields the above warning.

      I've seen instances where the actual disk usage of a directory is 25% of the cached value in INodeDirectory, which creates problems since the quota code uses this cached value to determine whether block write requests are permitted.

      This isn't easy to reproduce - I am able to (inconsistently) get HDFS into this state with a simple program which:

      1. Writes files into HDFS
      2. When a DSQuotaExceededException is encountered removes all files created in step 1
      3. Repeat step 1

      I'm going to try and come up with a more repeatable test case to reproduce this issue.

      Attachments

        1. QuotaTestSimple.java
          4 kB
          Alex Holmes
        2. hdfs-3061-branch-1.patch
          3 kB
          Kihwal Lee

        Issue Links

          Activity

            People

              kihwal Kihwal Lee
              alex.holmes Alex Holmes
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: