Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6535

HDFS quota update is wrong when file is appended

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.5.0
    • namenode
    • None
    • Reviewed

    Description

      when a file in the directory with Quota feature is appended, the cached disk consumption should be updated.
      But currently, the update is wrong.
      Use the uploaded UT to reproduce this bug.

      Attachments

        1. TestHDFSQuota.java
          5 kB
          George Wong
        2. HDFS-6535.patch
          6 kB
          George Wong
        3. HDFS-6535_v1.patch
          6 kB
          George Wong

        Issue Links

          Activity

            GWong George Wong added a comment -

            upload the ut to reproduce this issue

            GWong George Wong added a comment - upload the ut to reproduce this issue
            GWong George Wong added a comment -

            When appending a file, namenode does not update the disk consumption for the last block in the file. So, the cached disk consumption is less than the actual disk consumption.

            I am going to format a patch later. I have a fixed in my local branch currently.

            GWong George Wong added a comment - When appending a file, namenode does not update the disk consumption for the last block in the file. So, the cached disk consumption is less than the actual disk consumption. I am going to format a patch later. I have a fixed in my local branch currently.
            GWong George Wong added a comment -

            upload the patch to fix this issue.
            The patch works for current trunk.

            GWong George Wong added a comment - upload the patch to fix this issue. The patch works for current trunk.
            GWong George Wong added a comment -

            Hi brandonli, would you mind to assign this jira to me ?
            I am happy to follow up the fix of this.
            And could you review my patch? It is quite simple. Looking forward to your help.

            Thanks a lot.

            GWong George Wong added a comment - Hi brandonli , would you mind to assign this jira to me ? I am happy to follow up the fix of this. And could you review my patch? It is quite simple. Looking forward to your help. Thanks a lot.
            jingzhao Jing Zhao added a comment -

            Hi GWong, thanks for working on this. This is actually a bug from HDFS-6423. Your patch looks good to me. The only comment is that maybe we can move all the new unit tests into TestDiskspaceQuotaUpdate.java, where we should also change the number of DN and the replication factor to some number greater than 1.

            jingzhao Jing Zhao added a comment - Hi GWong , thanks for working on this. This is actually a bug from HDFS-6423 . Your patch looks good to me. The only comment is that maybe we can move all the new unit tests into TestDiskspaceQuotaUpdate.java, where we should also change the number of DN and the replication factor to some number greater than 1.
            GWong George Wong added a comment -

            Thanks Jing for your reply.

            OK. I will add more UT to address your comment.

            GWong George Wong added a comment - Thanks Jing for your reply. OK. I will add more UT to address your comment.
            GWong George Wong added a comment -

            change the patch to address Jing's comments.

            GWong George Wong added a comment - change the patch to address Jing's comments.
            GWong George Wong added a comment -

            jingzhao, when I am working on this issue, I find another issue about Quota update. Here is the filed jira
            https://issues.apache.org/jira/browse/HDFS-6575

            I think I may give a patch later.

            GWong George Wong added a comment - jingzhao , when I am working on this issue, I find another issue about Quota update. Here is the filed jira https://issues.apache.org/jira/browse/HDFS-6575 I think I may give a patch later.
            GWong George Wong added a comment -

            jingzhao, Could you please help me to review the updated patch? Thanks a lot.

            GWong George Wong added a comment - jingzhao , Could you please help me to review the updated patch? Thanks a lot.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12651642/HDFS-6535_v1.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 1 new or modified test files.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. There were no new javadoc warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7188//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7188//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651642/HDFS-6535_v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7188//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7188//console This message is automatically generated.
            jingzhao Jing Zhao added a comment -

            The patch looks good to me. +1. I will commit it shortly.

            jingzhao Jing Zhao added a comment - The patch looks good to me. +1. I will commit it shortly.
            jingzhao Jing Zhao added a comment -

            I've committed this to trunk and branch-2. Thanks for the contribution, GWong!

            jingzhao Jing Zhao added a comment - I've committed this to trunk and branch-2. Thanks for the contribution, GWong !
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-trunk-Commit #5743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5743/)
            HDFS-6535. HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5743/ ) HDFS-6535 . HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Yarn-trunk #590 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/590/)
            HDFS-6535. HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #590 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/590/ ) HDFS-6535 . HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Hdfs-trunk #1781 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1781/)
            HDFS-6535. HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1781 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1781/ ) HDFS-6535 . HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Mapreduce-trunk #1808 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1808/)
            HDFS-6535. HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1808 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1808/ ) HDFS-6535 . HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java

            People

              GWong George Wong
              GWong George Wong
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: