Description
Attachments
Attachments
- HDFS-7456-001.patch
- 4 kB
- Vinayakumar B
- HDFS-7456-002.patch
- 8 kB
- Vinayakumar B
- HDFS-7456-003.patch
- 14 kB
- Vinayakumar B
- HDFS-7456-004.patch
- 36 kB
- Vinayakumar B
- HDFS-7456-005.patch
- 37 kB
- Vinayakumar B
- HDFS-7456-006.patch
- 37 kB
- Vinayakumar B
- HDFS-7456-007.patch
- 40 kB
- Vinayakumar B
- HDFS-7456-008.patch
- 40 kB
- Vinayakumar B
- HDFS-7456-009.patch
- 40 kB
- Vinayakumar B
Issue Links
Activity
Rebased on trunk.
Added missing equals() for AclFeature.
Also added a unit test to verity the de-duplication.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12684625/HDFS-7456-002.patch
against trunk revision b36f292.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8896//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8896//console
This message is automatically generated.
I think this needs a rebase and some additional changes after HDFS-7454 went in. Here is a review of the current patch:
- AclFeature: Now that entries is an int[], we need to update equals and hashCode to use Arrays#equals and Arrays#hashCode.
- Interner: There are 2 calls to WeakReference#get. The first is done for a null check, and the second is to return the referent. If there is a garbage collection in between the 2 calls, then this could cause the method to return null, and the caller likely would hit a NullPointerException some time later.
- I'm looking back at the code for the Guava interner implementation that I was using in the original
HDFS-5620patch that we abandoned. That code is shown here: https://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/collect/Interners.java . I had been critical of the spinlock, but I now realize that this implementation has another nice benefit. It uses a dummy value in the map instead of the key wrapped in a WeakReference. This saves on memory, because it avoids the need for the extra layer of pointers going through WeakReference. As a side effect, it needs a spinlock, but in our usage, I expect little contention on the spin lock. We would only spin in the presence of multiple concurrent ACL modification operations. Reads wouldn't cause a spin. - The new tests create a few new ACLs and assert that we only create a new instance if needed. This is great. I'd also like to add tests that remove all references to an ACL and then assert that the instance count goes down. Of course, that could be a flaky test if we're relying on the GC, so that leads me to...
In general, my current thinking on this is that we should avoid implementing an interner at all and instead implement it as a reference-counted set, i.e. Map<AclEntry, Long>. The value type would need to be Long for agreement with the definition of inode ID. Unlike an interner, we'd need to hook code into both creation and removal instead of relying on weak references and GC to cover removal for us. Fortunately, that's a very easy thing for ACLs, because we already have the hook points we need. INodeWithAdditionalFields#addAclFeature can do the add, and INodeWithAdditionalFields#removeAclFeature can do the remove. We know both operations are done under the namesystem write lock, so I don't see any tricky concurrency challenges. This might end up being a larger total patch compared to the interner, but I think it's simpler code overall compared to reasoning through weak references and GC. As another benefit, we'll be able to write reliable tests of the removal case, because we'll be able to get a deterministic count out of the Map.
Let me know your thoughts on this. Thanks again for all of your recent ACL patches!
On a side note, the same techniques are applicable to XAttrFeature, though I don't know if anyone has enough experience with real-world xattr usage yet to tell if they tend to experience a high degree of duplication in the values like ACLs do.
Thanks cnauroth for the suggestions and comments.
Attaching the patch with ReferenceCountMap implementation. I chose this to avoid confusion as there were multiple RefereceMap classes seen.
Instead of holding a separate Long as value, I found easy to give the original referenced instance if that instance itself saved as value in map and it holds the reference count.
There was one more case where AclFeature reference count to be decreased. i.e. deletion of nodes. That also I have covered. Tests also added for all above cases.
Let me know your feedback on this patch.
On a side note, the same techniques are applicable to XAttrFeature, though I don't know if anyone has enough experience with real-world xattr usage yet to tell if they tend to experience a high degree of duplication in the values like ACLs do
I think, if this is required, then ReferenceCountMap added in v003 patch could be reused. And I feel this could be done in separate Jira though.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12685691/HDFS-7456-003.patch
against trunk revision 120e1de.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8947//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8947//console
This message is automatically generated.
Thank you for the patch, Vinay. Here are a few comments.
- In ReferenceCountMap#put, I see a way to avoid the instanceof pre-condition check and the 2 downcasts. Change the class type parameter to E extends ReferenceCounter and then change the type definition of the map to Map<E, E>. That way, the constraint is enforced at compile time.
- Please annotate the new class as Private.
- Thank you for covering the case of deleting inodes too!
- Can we please add some additional tests in FSAclBaseTest and TestAclWithSnapshot? On some of these, it might be easier to insert new assertions within existing tests instead of writing whole new tests. This is perhaps a bit paranoid, but here is the list of additional test cases I came up with, mostly focused on maintaining correct reference counts:
- Deleting a file with ACL decrements the reference count.
- Modifying a directory's ACL decrements the reference count of the old ACL and increments the reference count of the new ACL.
- Modifying a file's ACL decrements the reference count of the old ACL and increments the reference count of the new ACL.
- Deleting a snapshot where the root directory references an ACL decrements the reference count.
- Deleting a snapshot where a contained sub-directory references an ACL decrements the reference count.
- Deleting a snapshot where a contained file references an ACL decrements the reference count.
- After creating a snapshot, modifying the ACL of the root directory decrements the reference count of the old ACL and increments the reference count of the new ACL.
- After creating a snapshot, modifying the ACL of a contained sub-directory decrements the reference count of the old ACL and increments the reference count of the new ACL.
- After creating a snapshot, modifying the ACL of a contained file decrements the reference count of the old ACL and increments the reference count of the new ACL.
- After creating a snapshot that contains both directories and files with ACLs, and then deleting the original directory, check that the reference counts still show the snapshotted inodes are referencing their ACLs.
- Restart the NameNode. Assert that reference counts are the same before and after the restart.
- Save a new checkpoint and restart the NameNode. Assert that reference counts are the same before and after the restart.
Thanks cnauroth for the detailed tests and comments. I will try and post patch soon
Thanks again cnauroth for the proposed tests. It really helped a lot to fix issues related to snapshots.
I have added all tests proposed and addressed comments.
Please review the updated patch.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12686052/HDFS-7456-004.patch
against trunk revision 82707b4.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 287 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFSAcl
org.apache.hadoop.hdfs.TestLeaseRecovery2
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8976//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8976//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8976//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12686053/HDFS-7456-004.patch
against trunk revision 82707b4.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 287 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFSAcl
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8977//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8977//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8977//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12686181/HDFS-7456-005.patch
against trunk revision 437322a.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 40 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8990//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8990//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8990//console
This message is automatically generated.
Hi vinayrpet. The latest version is looking pretty good, though now it needs a rebase. Is ReferenceCountMap#getEntries unused? If so, can we please remove it?
Yhanks chris, Here is the rebased patch.
ReferenceCountMap#getEntries is being used in test. I have added @VisibileForTesting annotation.
Please review.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12688227/HDFS-7456-006.patch
against trunk revision efe6357.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.hdfs.server.namenode.TestINodeFile
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9089//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9089//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9089//console
This message is automatically generated.
Thanks again, Vinay. This is looking good, and the tests are very thorough. I have just a few more nitpicks.
In INodeAttributes, was it necessary to change the SnapshotCopy constructor? The new version has a slight disadvantage in that calling INode#getPermissionStatus constructs a new intermediate PermissionStatus, which then has to get translated back to long representation immediately. It will generate some additional garbage.
In FSAclBaseTest, perhaps it's more appropriate to rename startSize to currentSize, since it gets incremented in the test when new instances get created.
In the NameNode restart tests, we have the limitation that reference counts are not maintained between restarts, because the data is static. I think it's important to be able to test this though. Even though the reference counts won't be decremented during shutdown, we should know the expected delta of increments done on each feature during the subsequent startup. Can we add assertions for that?
Addressed above comments.
Thanks Again Chris for the suggestion of verifying the counts. While testing that found HDFS-7560, seems a serious bug.
Though I could verify counts for AClFeatures of all tests, but I intentionally reinited the cluster in testDeduplication in FSAclBaseTest to avoid failures due to restart of the cluster in some other tests ( ex: HDFS-7560) .
Please review.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12688553/HDFS-7456-007.patch
against trunk revision 24ee9e3.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9103//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9103//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9103//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12688565/HDFS-7456-008.patch
against trunk revision 8f5522e.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9104//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9104//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9104//console
This message is automatically generated.
In the v008 patch, I spotted one minor issue. There is a test assertion with a message about "doubling" the count, but really it's tripling the count:
assertEquals("ReferenceCount After Restart should be doubled",
before * 3, aclFeature.getRefCount());
I'm +1 for the patch after correcting that and getting one more Jenkins run. Vinay, please feel free to commit after that.
I also did some additional manual testing on this. I set a default ACL on a directory. Then, I created 100 sub-directories and 100 files in that directory. As expected, after a full GC, the heap histogram showed 201 instances of AclFeature. I then applied the patch, restarted the NameNode, triggered a full GC, and checked the heap histogram again. This time, I saw only 3 instances of AclFeature: 1 for the first directory, 1 shared across all child directories, and 1 shared across all child files.
Nice work, Vinay! Thank you.
Thanks cnauroth for the additional manual verification. Here is the updated patch.
After jenkins run I will commit this.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12688786/HDFS-7456-009.patch
against trunk revision fdf042d.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9116//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9116//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9116//console
This message is automatically generated.
Failures and findbugs are not related to patch.
I will commit the patch soon.
Thanks cnauroth for all the reviews and wonderful suggestions.
Committed v009 patch to trunk and branch-2.
FAILURE: Integrated in Hadoop-trunk-Commit #6778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6778/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #50 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/50/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
SUCCESS: Integrated in Hadoop-Yarn-trunk #784 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/784/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #47 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/47/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1982 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1982/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #51 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/51/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2001/)
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode (Contributed by Vinayakumar B) (vinayakumarb: rev 50ae1a6664a92619aa683d2a864d0da9fb4af026)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestAclWithSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileContextAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ReferenceCountMap.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSAcl.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
Attached the patch, as discussed in here.
Simple Interner class has been added, which interns the entries without any explicit lock by assuming that all operations are done under external lock.
hi cnauroth and wheat9, can you take a look at the patch.?
Thanks.