Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      After HDFS-6763, the quota on the standby namenode is not being updated until it transitions to active. This causes the jmx and the web ui files and dir count to be uninitialized or unupdated. In some cases it shows a negative number.

      It is because the legacy way of getting the inode count, which existed since before the creation of inode table. It relies on the root inode's quota being properly updated. We can make it simply return the size of the inode table.

      Attachments

        1. HDFS-9396.patch
          0.7 kB
          Kihwal Lee
        2. HDFS-9396.v2.patch
          4 kB
          Kihwal Lee

        Issue Links

        Activity

          kihwal Kihwal Lee added a comment -
            long totalInodes() {
              readLock();
              try {
                return rootDir.getDirectoryWithQuotaFeature().getSpaceConsumed()
                    .getNameSpace();
              } finally {
                readUnlock();
              }
            }
          

          It can simply do without locking.

            return inodeMap.size();
          
          kihwal Kihwal Lee added a comment - long totalInodes() { readLock(); try { return rootDir.getDirectoryWithQuotaFeature().getSpaceConsumed() .getNameSpace(); } finally { readUnlock(); } } It can simply do without locking. return inodeMap.size();
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 3m 25s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_60
          +1 compile 0m 33s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 17s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 0s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 42s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_60
          +1 javac 0m 37s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_79
          +1 javac 0m 34s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 0s the patch passed with JDK v1.7.0_79
          -1 unit 60m 29s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 30m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          112m 36s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.TestDFSShell
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.TestCacheDirectives
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771121/HDFS-9396.patch
          JIRA Issue HDFS-9396
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 72a4dd029485 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / ef926b2
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13438/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 34MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13438/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 3m 25s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_60 +1 compile 0m 33s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 17s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 0s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 42s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_60 +1 javac 0m 37s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_79 +1 javac 0m 34s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 0s the patch passed with JDK v1.7.0_79 -1 unit 60m 29s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 30m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 112m 36s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771121/HDFS-9396.patch JIRA Issue HDFS-9396 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 72a4dd029485 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / ef926b2 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13438/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 34MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13438/console This message was automatically generated.
          kihwal Kihwal Lee added a comment -

          Added a unit test.

          kihwal Kihwal Lee added a comment - Added a unit test.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 53s trunk passed
          +1 compile 0m 46s trunk passed with JDK v1.8.0_60
          +1 compile 0m 40s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 20s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 22s trunk passed
          +1 javadoc 1m 34s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 27s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.8.0_60
          +1 javac 0m 46s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_79
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 38s the patch passed
          +1 javadoc 1m 32s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 26s the patch passed with JDK v1.7.0_79
          -1 unit 80m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 71m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 28s Patch generated 56 ASF License warnings.
          178m 51s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771608/HDFS-9396.v2.patch
          JIRA Issue HDFS-9396
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 96e550b5d9ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / c99925d
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13463/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13463/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 53s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_60 +1 compile 0m 40s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 20s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 22s trunk passed +1 javadoc 1m 34s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 27s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 50s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_60 +1 javac 0m 46s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_79 +1 javac 0m 40s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 38s the patch passed +1 javadoc 1m 32s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 26s the patch passed with JDK v1.7.0_79 -1 unit 80m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 71m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 28s Patch generated 56 ASF License warnings. 178m 51s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771608/HDFS-9396.v2.patch JIRA Issue HDFS-9396 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 96e550b5d9ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / c99925d Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13463/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13463/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13463/console This message was automatically generated.
          kihwal Kihwal Lee added a comment -
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.TestHFlush
          Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.082 sec - in org.apache.hadoop.hdfs.TestHFlush
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.543 sec - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.408 sec - in org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.382 sec - in org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.374 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.013 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          testCircularLinkedListWrites(org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes)  Time elapsed: 126.826 sec  <<< FAILURE!
          java.lang.AssertionError: Some writers didn't complete in expected runtime! Current writer state:[Circular Writer:
          	 directory: /test-1
          	 target length: 50
          	 current item: 1
          	 done: false
          ] expected:<0> but was:<1>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes.testCircularLinkedListWrites(TestSeveralNameNodes.java:90)
          
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.884 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 1.192 sec - in org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.TestEncryptedTransfer
          Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.527 sec - in org.apache.hadoop.hdfs.TestEncryptedTransfer
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.674 sec - in org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          

          When run on my machine, all failed test cases but TestSeveralNameNodes pass. This is not related to the patch and will be addressed in HDFS-9376.

          kihwal Kihwal Lee added a comment - ------------------------------------------------------- T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestHFlush Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.082 sec - in org.apache.hadoop.hdfs.TestHFlush Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.543 sec - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.408 sec - in org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.382 sec - in org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.374 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.013 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes testCircularLinkedListWrites(org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes) Time elapsed: 126.826 sec <<< FAILURE! java.lang.AssertionError: Some writers didn't complete in expected runtime! Current writer state:[Circular Writer: directory: /test-1 target length: 50 current item: 1 done: false ] expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes.testCircularLinkedListWrites(TestSeveralNameNodes.java:90) Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.884 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache Tests run: 11, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 1.192 sec - in org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestEncryptedTransfer Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.527 sec - in org.apache.hadoop.hdfs.TestEncryptedTransfer Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.674 sec - in org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser When run on my machine, all failed test cases but TestSeveralNameNodes pass. This is not related to the patch and will be addressed in HDFS-9376 .
          daryn Daryn Sharp added a comment -

          +1 Terrible source of confusion and alarm to have -10k files and directories.

          daryn Daryn Sharp added a comment - +1 Terrible source of confusion and alarm to have -10k files and directories.
          kihwal Kihwal Lee added a comment -

          Thanks for the review, Daryn. I've committed this to trunk and branch-2.

          kihwal Kihwal Lee added a comment - Thanks for the review, Daryn. I've committed this to trunk and branch-2.
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8794 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8794/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8794 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8794/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2599 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2599/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2599 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2599/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1395 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1395/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1395 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1395/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #671 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/671/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #671 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/671/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #659 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/659/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #659 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/659/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #598 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/598/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #598 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/598/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2537 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2537/)
          HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2537 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2537/ ) HDFS-9396 . Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

          People

            kihwal Kihwal Lee
            kihwal Kihwal Lee
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack