Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
After HDFS-6763, the quota on the standby namenode is not being updated until it transitions to active. This causes the jmx and the web ui files and dir count to be uninitialized or unupdated. In some cases it shows a negative number.
It is because the legacy way of getting the inode count, which existed since before the creation of inode table. It relies on the root inode's quota being properly updated. We can make it simply return the size of the inode table.
Attachments
Attachments
- HDFS-9396.patch
- 0.7 kB
- Kihwal Lee
- HDFS-9396.v2.patch
- 4 kB
- Kihwal Lee
Issue Links
- is broken by
-
HDFS-6763 Initialize file system-wide quota once on transitioning to active
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 7s | docker + precommit patch detected. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | mvninstall | 3m 25s | trunk passed |
+1 | compile | 0m 38s | trunk passed with JDK v1.8.0_60 |
+1 | compile | 0m 33s | trunk passed with JDK v1.7.0_79 |
+1 | checkstyle | 0m 17s | trunk passed |
+1 | mvneclipse | 0m 14s | trunk passed |
-1 | findbugs | 2m 3s | hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 15s | trunk passed with JDK v1.8.0_60 |
+1 | javadoc | 2m 0s | trunk passed with JDK v1.7.0_79 |
+1 | mvninstall | 0m 42s | the patch passed |
+1 | compile | 0m 37s | the patch passed with JDK v1.8.0_60 |
+1 | javac | 0m 37s | the patch passed |
+1 | compile | 0m 34s | the patch passed with JDK v1.7.0_79 |
+1 | javac | 0m 34s | the patch passed |
+1 | checkstyle | 0m 16s | the patch passed |
+1 | mvneclipse | 0m 14s | the patch passed |
+1 | whitespace | 0m 0s | Patch has no whitespace issues. |
+1 | findbugs | 2m 15s | the patch passed |
+1 | javadoc | 1m 12s | the patch passed with JDK v1.8.0_60 |
+1 | javadoc | 2m 0s | the patch passed with JDK v1.7.0_79 |
-1 | unit | 60m 29s | hadoop-hdfs in the patch failed with JDK v1.8.0_60. |
-1 | unit | 30m 31s | hadoop-hdfs in the patch failed with JDK v1.7.0_79. |
+1 | asflicense | 0m 20s | Patch does not generate ASF License warnings. |
112m 36s |
Reason | Tests |
---|---|
JDK v1.8.0_60 Failed junit tests | hadoop.hdfs.TestDFSShell |
hadoop.hdfs.server.blockmanagement.TestBlockManager | |
hadoop.hdfs.server.namenode.TestCacheDirectives | |
JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.server.namenode.TestStartup |
hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 9s | docker + precommit patch detected. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 3m 53s | trunk passed |
+1 | compile | 0m 46s | trunk passed with JDK v1.8.0_60 |
+1 | compile | 0m 40s | trunk passed with JDK v1.7.0_79 |
+1 | checkstyle | 0m 20s | trunk passed |
+1 | mvneclipse | 0m 17s | trunk passed |
+1 | findbugs | 2m 22s | trunk passed |
+1 | javadoc | 1m 34s | trunk passed with JDK v1.8.0_60 |
+1 | javadoc | 2m 27s | trunk passed with JDK v1.7.0_79 |
+1 | mvninstall | 0m 50s | the patch passed |
+1 | compile | 0m 46s | the patch passed with JDK v1.8.0_60 |
+1 | javac | 0m 46s | the patch passed |
+1 | compile | 0m 40s | the patch passed with JDK v1.7.0_79 |
+1 | javac | 0m 40s | the patch passed |
+1 | checkstyle | 0m 21s | the patch passed |
+1 | mvneclipse | 0m 17s | the patch passed |
+1 | whitespace | 0m 0s | Patch has no whitespace issues. |
+1 | findbugs | 2m 38s | the patch passed |
+1 | javadoc | 1m 32s | the patch passed with JDK v1.8.0_60 |
+1 | javadoc | 2m 26s | the patch passed with JDK v1.7.0_79 |
-1 | unit | 80m 43s | hadoop-hdfs in the patch failed with JDK v1.8.0_60. |
-1 | unit | 71m 38s | hadoop-hdfs in the patch failed with JDK v1.7.0_79. |
-1 | asflicense | 0m 28s | Patch generated 56 ASF License warnings. |
178m 51s |
Reason | Tests |
---|---|
JDK v1.8.0_60 Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
hadoop.hdfs.server.datanode.TestDirectoryScanner | |
hadoop.hdfs.TestEncryptedTransfer | |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer | |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | |
JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.TestHFlush |
hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
hadoop.hdfs.server.namenode.TestDeadDatanode | |
hadoop.hdfs.server.datanode.TestDirectoryScanner | |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer | |
hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
This message was automatically generated.
------------------------------------------------------- T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestHFlush Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.082 sec - in org.apache.hadoop.hdfs.TestHFlush Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.543 sec - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.408 sec - in org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.382 sec - in org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.374 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.013 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes testCircularLinkedListWrites(org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes) Time elapsed: 126.826 sec <<< FAILURE! java.lang.AssertionError: Some writers didn't complete in expected runtime! Current writer state:[Circular Writer: directory: /test-1 target length: 50 current item: 1 done: false ] expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes.testCircularLinkedListWrites(TestSeveralNameNodes.java:90) Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.884 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache Tests run: 11, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 1.192 sec - in org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestEncryptedTransfer Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.527 sec - in org.apache.hadoop.hdfs.TestEncryptedTransfer Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.674 sec - in org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
When run on my machine, all failed test cases but TestSeveralNameNodes pass. This is not related to the patch and will be addressed in HDFS-9376.
+1 Terrible source of confusion and alarm to have -10k files and directories.
Thanks for the review, Daryn. I've committed this to trunk and branch-2.
FAILURE: Integrated in Hadoop-trunk-Commit #8794 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8794/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2599 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2599/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
FAILURE: Integrated in Hadoop-Yarn-trunk #1395 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1395/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #671 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/671/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #659 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/659/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #598 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/598/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Hdfs-trunk #2537 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2537/)
HDFS-9396. Total files and directories on jmx and web UI on standby is (kihwal: rev a2a104f3d30182953db6690304146f0a582ed241)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAMetrics.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
It can simply do without locking.
return inodeMap.size();