Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8900

Compact XAttrs to optimize memory footprint.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The config key "dfs.namenode.fs-limits.max-xattr-size" can no longer be set to a value of 0 (previously used to indicate unlimited) or a value greater than 32KB. This is a constraint on xattr size similar to many local filesystems.

      Description

      private final ImmutableList<XAttr> xAttrs;
      

      Currently we use above in XAttrFeature, it's not efficient from memory point of view, since ImmutableList and XAttr have object memory overhead, and each object has memory alignment.

      We can use a byte[] in XAttrFeature and do some compact in XAttr.

      1. HDFS-8900.001.patch
        32 kB
        Yi Liu
      2. HDFS-8900.002.patch
        37 kB
        Yi Liu
      3. HDFS-8900.003.patch
        39 kB
        Yi Liu
      4. HDFS-8900.004.patch
        41 kB
        Yi Liu
      5. HDFS-8900.005.patch
        41 kB
        Yi Liu

        Issue Links

          Activity

          Hide
          hitliuyi Yi Liu added a comment - - edited

          This optimization:

          • For each XAttr, can save: 12bytes (object overhead) + 2bytes (compact), memory alignment of value
          • For each XAttrFeature, can save: (ImmutableList object size) + num * (XAttr saved).

          So for each XAttrFeature (associated with an INodeFile or INodeDirectory), we can save (bytes): (ImmutableList object size) + n * (14 + memory alignment of value).

          • If n == 1, (ImmutableList object size) = 12 + 4 (bytes). Totally 30 bytes.
          • if n > 1, ImmutableList is RegularImmutableList, so (ImmutableList object size) = 12 + 4 + 4 + 12 + n * 4 + object alignment. Totally 36 + n * (18 + memory alignment of value) bytes.

          We can see the improvement is significant especially there is more than one XAttr in XAttrFeature.

          The patch also makes code logic more simper, and has no impact on NN performance.

          Besides, the xattrMaxSize is used to restrict user/trust namespace xattrs, I add a hard limit for max size. It's reasonable and make Xattr more controllable.

          Show
          hitliuyi Yi Liu added a comment - - edited This optimization: For each XAttr, can save: 12bytes (object overhead) + 2bytes (compact), memory alignment of value For each XAttrFeature, can save: (ImmutableList object size) + num * (XAttr saved). So for each XAttrFeature (associated with an INodeFile or INodeDirectory), we can save (bytes): (ImmutableList object size) + n * (14 + memory alignment of value). If n == 1 , (ImmutableList object size) = 12 + 4 (bytes). Totally 30 bytes. if n > 1 , ImmutableList is RegularImmutableList, so (ImmutableList object size) = 12 + 4 + 4 + 12 + n * 4 + object alignment. Totally 36 + n * (18 + memory alignment of value) bytes . We can see the improvement is significant especially there is more than one XAttr in XAttrFeature. The patch also makes code logic more simper, and has no impact on NN performance. Besides, the xattrMaxSize is used to restrict user/trust namespace xattrs, I add a hard limit for max size. It's reasonable and make Xattr more controllable.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Yi. Change looks great, seems like a nice simple dictionary encoding scheme.

          The hard limit changes are incompatible, but I see why it's required to limit the length to a 2B field. Other filesystems have a limit which isn't even configurable, so seems okay. Let's flag as such and add a release note. Maybe this should only go into trunk as a result.

          Hard limit related:

          • Could we exit on setting an xattr size bigger than the hard limit, rather than doing a silent min? We should mention this new hard limit somewhere as well.
          • Any comment on the max size supported by other filesystems like ext4 or btrfs, for reference as to what's reasonable here?
          • Typo in FSDirectory: "should" -> "should be". Also the line below it says "unlimited" but that'll never get triggered now.
          • Regarding configuration, we could simplify by just using the hard limit. Admins would still have the option of disabling xattrs entirely; is there really any value in being able to set something smaller than 32KB? This would definitely make it a trunk change.

          More review comments:

          • FSDirectory#addToInodeMap, do we need that new return?
          • A bit out of scope and so optional, but I think everywhere we say "prefixName" we really want to say "prefixedName" because "prefixName" sounds more like the name of the prefix rather than a name with a prefix.
          • Some unrelated import changes in FSNamesystemLock and INodeAttributeProvider
          • XAttrFeature has an extra import
          • What's the reason for switching from ImmutableList to List in some places? The switch is also not complete, since I still see some usages of ImmutableList. I remember we liked ImmutableList originally since it made the need to set very explicit.
          • Mind adding Javadoc for SerialNumberMap, and an interface audience private annotation?
          • XAttrsFormat's class javadoc goes over 80 chars, could use interface audience private also.
          • Can we add an IllegalStateException to SerialNumberMap#get(T) for Integer overflow? Also there's the case that the int from the map doesn't fit in the 29 bits in XAttrFormat; check that in XAttrsFormattoBytes?
          • Consider also renaming XAttrsFormat to XAttrFormat, so it's named like XAttrStorage
          • Is it worthwhile to do the same dictionary encoding for the FSImage as well? If the # xattrs is large enough to affect memory footprint, it'd also affect loading times which can already be minutes. Can be a follow-on JIRA for sure.
          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Yi. Change looks great, seems like a nice simple dictionary encoding scheme. The hard limit changes are incompatible, but I see why it's required to limit the length to a 2B field. Other filesystems have a limit which isn't even configurable, so seems okay. Let's flag as such and add a release note. Maybe this should only go into trunk as a result. Hard limit related: Could we exit on setting an xattr size bigger than the hard limit, rather than doing a silent min? We should mention this new hard limit somewhere as well. Any comment on the max size supported by other filesystems like ext4 or btrfs, for reference as to what's reasonable here? Typo in FSDirectory: "should" -> "should be". Also the line below it says "unlimited" but that'll never get triggered now. Regarding configuration, we could simplify by just using the hard limit. Admins would still have the option of disabling xattrs entirely; is there really any value in being able to set something smaller than 32KB? This would definitely make it a trunk change. More review comments: FSDirectory#addToInodeMap, do we need that new return? A bit out of scope and so optional, but I think everywhere we say "prefixName" we really want to say "prefixedName" because "prefixName" sounds more like the name of the prefix rather than a name with a prefix. Some unrelated import changes in FSNamesystemLock and INodeAttributeProvider XAttrFeature has an extra import What's the reason for switching from ImmutableList to List in some places? The switch is also not complete, since I still see some usages of ImmutableList. I remember we liked ImmutableList originally since it made the need to set very explicit. Mind adding Javadoc for SerialNumberMap, and an interface audience private annotation? XAttrsFormat's class javadoc goes over 80 chars, could use interface audience private also. Can we add an IllegalStateException to SerialNumberMap#get(T) for Integer overflow? Also there's the case that the int from the map doesn't fit in the 29 bits in XAttrFormat; check that in XAttrsFormattoBytes? Consider also renaming XAttrsFormat to XAttrFormat, so it's named like XAttrStorage Is it worthwhile to do the same dictionary encoding for the FSImage as well? If the # xattrs is large enough to affect memory footprint, it'd also affect loading times which can already be minutes. Can be a follow-on JIRA for sure.
          Hide
          andrew.wang Andrew Wang added a comment -

          I talked with Colin P. McCabe about compatibility offline, we had a few thoughts:

          • Unlimited size xattrs are an anti-pattern and not that useful. Same for very large xattrs in general.
          • For branch-2, one compatible approach is to size the length field based on the max size config, up to a full 4-byte signed integer. This is what "unlimited" currently means anyway. Then, do a trunk follow-on to just use a 2-byte length and maybe get rid of the max size config in favor of the hard limit.
          • Practically speaking, it's unlikely anyone's ever changed the max size, so the impact of the incompatibility is limited. Patch might be fine for branch-2 as is.
          Show
          andrew.wang Andrew Wang added a comment - I talked with Colin P. McCabe about compatibility offline, we had a few thoughts: Unlimited size xattrs are an anti-pattern and not that useful. Same for very large xattrs in general. For branch-2, one compatible approach is to size the length field based on the max size config, up to a full 4-byte signed integer. This is what "unlimited" currently means anyway. Then, do a trunk follow-on to just use a 2-byte length and maybe get rid of the max size config in favor of the hard limit. Practically speaking, it's unlikely anyone's ever changed the max size, so the impact of the incompatibility is limited. Patch might be fine for branch-2 as is.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          Thanks Andrew Wang for the quick review and detailed comments (Actually I was planing to ask you for help if you were not here , since this is one of your specialistic areas) !
          I will update the patch and reply you later today.

          Show
          hitliuyi Yi Liu added a comment - - edited Thanks Andrew Wang for the quick review and detailed comments (Actually I was planing to ask you for help if you were not here , since this is one of your specialistic areas) ! I will update the patch and reply you later today.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          Thanks Andrew Wang, I update the patch for your comments.

          About hard limit.

          I talked with Colin Patrick McCabe about compatibility offline ....

          Yes, I agree you and Colin. In fact, when I added the hard limit, the first thing I considered is the compatibility. I did it based on two reasons: 1). as you said too, it's unlikely anyone's ever changed the max size. 2). The max limit is to restrict user's (user/trusted) namespace xattrs, and it doesn't break the existing HDFS features.
          I felt it was OK to do this modification on trunk and branch-2 directly (Certainly we can still use 4 bytes, but I wanted to save 2 bytes from it ).
          I see you agree with the hard limit generally, one thing is do we need to only have the hard limit in trunk? How about do it in the branch-2 if we think it's fine?

          Could we exit on setting an xattr size bigger than the hard limit, rather than doing a silent min? We should mention this new hard limit somewhere as well.

          Sure, updated in the new patch. User will see the hard limit from error msg. I updated the description of the xattr max size in hdfs-default.xml and mentioned the hard limit too.

          Any comment on the max size supported by other filesystems like ext4 or btrfs, for reference as to what's reasonable here?

          It's a long time since I read xattr of ext4 and other fs last time. I remember some fs have limit about the xattr length, and some fs don't have. It's detail, small difference should be OK.

          Typo in FSDirectory: "should" -> "should be". Also the line below it says "unlimited" but that'll never get triggered now.

          sure, I forgot to remove the unlimited...

          Regarding configuration, we could simplify by just using the hard limit. Admins would still have the option of disabling xattrs entirely; is there really any value in being able to set something smaller than 32KB? This would definitely make it a trunk change.

          Agree, How about doing it in a follow-on if we want later?

          More review comments:

          FSDirectory#addToInodeMap, do we need that new return?

          right, no need. I removed it. Maybe I added it suddenly

          A bit out of scope and so optional, but I think everywhere we say "prefixName" we really want to say "prefixedName" because "prefixName" sounds more like the name of the prefix rather than a name with a prefix.

          Good idea, prefixedName is much more better, I updated all those in the new patch.

          Some unrelated import changes in FSNamesystemLock and INodeAttributeProvider

          OK, I reverted this modification since they are unrelated (I saw them, I was thinking to remove them).

          XAttrFeature has an extra import

          fixed

          What's the reason for switching from ImmutableList to List in some places? The switch is also not complete, since I still see some usages of ImmutableList. I remember we liked ImmutableList originally since it made the need to set very explicit.

          I think originally we use ImmutableList is mainly because it's immutable, and we keep it in XAttrFeature, we don't want outside modification affects it. Now it's packed to byte[] in XAttrFeature, so no need immutable list.
          I use ArrayList instead of ImmutableList is because when building ImmutableList, it needs additional list copy (from an internal ArrayList). Then the performance is a bit better.
          I missed to switch one ImmutableList in XAttrStorage and fixed it now.

          Mind adding Javadoc for SerialNumberMap, and an interface audience private annotation?
          XAttrsFormat's class javadoc goes over 80 chars, could use interface audience private also.

          Sure, updated them. One thing is XAttrFormat is package-private, so no need to add audience private annotation for it.

          Can we add an IllegalStateException to SerialNumberMap#get(T) for Integer overflow? Also there's the case that the int from the map doesn't fit in the 29 bits in XAttrFormat; check that in XAttrsFormattoBytes?

          Sure, I also add the check in XAttrFormat#toBytes. Actually I want to create a follow-on to restrict the total number of xattr names for user's (user/trust) xattrs. For HDFS kernel, the number of xattr names are less than 10 currently, but if users create many various xattrs (maybe by mistake), then it will cause unexpected behavior.

          Consider also renaming XAttrsFormat to XAttrFormat, so it's named like XAttrStorage

          Good idea.

          Is it worthwhile to do the same dictionary encoding for the FSImage as well? If the # xattrs is large enough to affect memory footprint, it'd also affect loading times which can already be minutes. Can be a follow-on JIRA for sure.

          Actually we already have this compaction for XAttr in FSImage, please see XAttrCompactProto in fsimage.proto.

          Show
          hitliuyi Yi Liu added a comment - - edited Thanks Andrew Wang , I update the patch for your comments. About hard limit. I talked with Colin Patrick McCabe about compatibility offline .... Yes, I agree you and Colin. In fact, when I added the hard limit, the first thing I considered is the compatibility. I did it based on two reasons: 1). as you said too, it's unlikely anyone's ever changed the max size. 2). The max limit is to restrict user's (user/trusted) namespace xattrs, and it doesn't break the existing HDFS features. I felt it was OK to do this modification on trunk and branch-2 directly (Certainly we can still use 4 bytes, but I wanted to save 2 bytes from it ). I see you agree with the hard limit generally, one thing is do we need to only have the hard limit in trunk? How about do it in the branch-2 if we think it's fine? Could we exit on setting an xattr size bigger than the hard limit, rather than doing a silent min? We should mention this new hard limit somewhere as well. Sure, updated in the new patch. User will see the hard limit from error msg. I updated the description of the xattr max size in hdfs-default.xml and mentioned the hard limit too. Any comment on the max size supported by other filesystems like ext4 or btrfs, for reference as to what's reasonable here? It's a long time since I read xattr of ext4 and other fs last time. I remember some fs have limit about the xattr length, and some fs don't have. It's detail, small difference should be OK. Typo in FSDirectory: "should" -> "should be". Also the line below it says "unlimited" but that'll never get triggered now. sure, I forgot to remove the unlimited... Regarding configuration, we could simplify by just using the hard limit. Admins would still have the option of disabling xattrs entirely; is there really any value in being able to set something smaller than 32KB? This would definitely make it a trunk change. Agree, How about doing it in a follow-on if we want later? More review comments: FSDirectory#addToInodeMap, do we need that new return? right, no need. I removed it. Maybe I added it suddenly A bit out of scope and so optional, but I think everywhere we say "prefixName" we really want to say "prefixedName" because "prefixName" sounds more like the name of the prefix rather than a name with a prefix. Good idea, prefixedName is much more better, I updated all those in the new patch. Some unrelated import changes in FSNamesystemLock and INodeAttributeProvider OK, I reverted this modification since they are unrelated (I saw them, I was thinking to remove them). XAttrFeature has an extra import fixed What's the reason for switching from ImmutableList to List in some places? The switch is also not complete, since I still see some usages of ImmutableList. I remember we liked ImmutableList originally since it made the need to set very explicit. I think originally we use ImmutableList is mainly because it's immutable, and we keep it in XAttrFeature , we don't want outside modification affects it. Now it's packed to byte[] in XAttrFeature , so no need immutable list. I use ArrayList instead of ImmutableList is because when building ImmutableList, it needs additional list copy (from an internal ArrayList). Then the performance is a bit better. I missed to switch one ImmutableList in XAttrStorage and fixed it now. Mind adding Javadoc for SerialNumberMap, and an interface audience private annotation? XAttrsFormat's class javadoc goes over 80 chars, could use interface audience private also. Sure, updated them. One thing is XAttrFormat is package-private , so no need to add audience private annotation for it. Can we add an IllegalStateException to SerialNumberMap#get(T) for Integer overflow? Also there's the case that the int from the map doesn't fit in the 29 bits in XAttrFormat; check that in XAttrsFormattoBytes? Sure, I also add the check in XAttrFormat#toBytes. Actually I want to create a follow-on to restrict the total number of xattr names for user's (user/trust) xattrs. For HDFS kernel, the number of xattr names are less than 10 currently, but if users create many various xattrs (maybe by mistake), then it will cause unexpected behavior. Consider also renaming XAttrsFormat to XAttrFormat, so it's named like XAttrStorage Good idea. Is it worthwhile to do the same dictionary encoding for the FSImage as well? If the # xattrs is large enough to affect memory footprint, it'd also affect loading times which can already be minutes. Can be a follow-on JIRA for sure. Actually we already have this compaction for XAttr in FSImage, please see XAttrCompactProto in fsimage.proto.
          Hide
          hitliuyi Yi Liu added a comment -

          (I don't know why the Jenkins is not triggered after submitting the patch, now I trigger the Jenkins manually.)

          Show
          hitliuyi Yi Liu added a comment - (I don't know why the Jenkins is not triggered after submitting the patch, now I trigger the Jenkins manually.)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 2s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 27s The applied patch generated 7 new checkstyle issues (total was 516, now 517).
          -1 whitespace 0m 1s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 34s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 8s Pre-build of native portion
          -1 hdfs tests 128m 28s Tests failed in hadoop-hdfs.
              173m 42s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.namenode.TestStartup
          Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751731/HDFS-8900.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 22de7c1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12066/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12066/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 27s The applied patch generated 7 new checkstyle issues (total was 516, now 517). -1 whitespace 0m 1s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 34s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 native 3m 8s Pre-build of native portion -1 hdfs tests 128m 28s Tests failed in hadoop-hdfs.     173m 42s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.namenode.TestStartup Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751731/HDFS-8900.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22de7c1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12066/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12066/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12066/console This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Latest rev LGTM, +1 pending. I'm okay with this in branch-2 as discussed above, we can remove the max size config in a trunk patch.

          Colin P. McCabe cool with you too?

          Show
          andrew.wang Andrew Wang added a comment - Latest rev LGTM, +1 pending. I'm okay with this in branch-2 as discussed above, we can remove the max size config in a trunk patch. Colin P. McCabe cool with you too?
          Hide
          cmccabe Colin P. McCabe added a comment -

          Sounds good to me. Thanks, guys.

          Show
          cmccabe Colin P. McCabe added a comment - Sounds good to me. Thanks, guys.
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Andrew Wang and Colin P. McCabe!
          I will create a follow-on JIRA targeted on trunk to remove the max size config.

          Failure of TestStartup.testXattrConfiguration is related, the new patch fixes it, and also fixes the checkstyle and whitespace. The findbugs actually don't exist.

          Show
          hitliuyi Yi Liu added a comment - Thanks Andrew Wang and Colin P. McCabe ! I will create a follow-on JIRA targeted on trunk to remove the max size config. Failure of TestStartup.testXattrConfiguration is related, the new patch fixes it, and also fixes the checkstyle and whitespace. The findbugs actually don't exist.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 46s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 24s The applied patch generated 2 new checkstyle issues (total was 517, now 513).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 32s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 9s Pre-build of native portion
          -1 hdfs tests 190m 51s Tests failed in hadoop-hdfs.
              235m 57s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.blockmanagement.TestNodeCount



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751974/HDFS-8900.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / feaf034
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12081/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12081/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 46s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 24s The applied patch generated 2 new checkstyle issues (total was 517, now 513). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 32s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 190m 51s Tests failed in hadoop-hdfs.     235m 57s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751974/HDFS-8900.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / feaf034 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12081/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12081/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12081/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Yi Lu for working on this...

          I have a little concern about GC problem. since now Xattr.value is a byte[].

          In the old design, When we update a Xattr, we doesn't changes other xattrs. In the proposed design, the values of all xattrs are combined in one packed byte[] .

          The packed byte[] could be large. Everytime we change one xattr, we have to dispose the packed byte[] and create a new one, even though other xattrs are not updated.It may be not a problem when set an xattr for a file. But when we recursively set for a directory, It burdens the GC. The max size of the packed byte[] is 32KB * 32 = 1MB. It's unlikely to reach that. But such big sequential memory allocation could cause full GC when heap is fragmented.

          So, maybe we could use byte array or use off-heap memory(may slow-down operations..?) for Xattrs.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Yi Lu for working on this... I have a little concern about GC problem. since now Xattr.value is a byte[]. In the old design, When we update a Xattr, we doesn't changes other xattrs. In the proposed design, the values of all xattrs are combined in one packed byte[] . The packed byte[] could be large. Everytime we change one xattr, we have to dispose the packed byte[] and create a new one, even though other xattrs are not updated.It may be not a problem when set an xattr for a file. But when we recursively set for a directory, It burdens the GC. The max size of the packed byte[] is 32KB * 32 = 1MB. It's unlikely to reach that. But such big sequential memory allocation could cause full GC when heap is fragmented. So, maybe we could use byte array or use off-heap memory(may slow-down operations..?) for Xattrs.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          So, maybe we could use byte array or use off-heap memory(may slow-down operations..?) for Xattrs.
          So, maybe we could recyle byte array or use off-heap memory for Xattrs.

          Show
          brahmareddy Brahma Reddy Battula added a comment - So, maybe we could use byte array or use off-heap memory(may slow-down operations..?) for Xattrs. So, maybe we could recyle byte array or use off-heap memory for Xattrs.
          Hide
          hitliuyi Yi Liu added a comment -

          The test failures are not related. There is no findbugs actually.

          Show
          hitliuyi Yi Liu added a comment - The test failures are not related. There is no findbugs actually.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          The max size of the packed byte[] is 32KB * 32 = 1MB...
          But such big sequential memory allocation could cause full GC when heap is fragmented.

          We can say this would never happen in real case. Typical the xattr value size is very small (several bytes to tens of bytes, much smaller than 32KB, this is also the design goal of xattr, for system/security namespace xattr used by HDFS core, the size are very small today; if applications use user/trusted xattr to store many large size xattrs, the memory of NN itself will have serious problem, for totally 1MB size xattrs of an INode as in your example, 1TB memory is required if there are only 1MB number Inodes). In real case, even for HDFS transparent encryption which uses xattr to store lots of information (see FileEncryptionInfo), the xattr value size is only 48bytes.

          Of course the applications may use the xattr wrongly and use it to many store large size (several KB to tens of KB) xattrs, to prevent this, I will add a threshold not to pack those large size xattrs and still store them separately in a list as original. For small size xattr, we will pack them. The threshold is chosen to be 1KB, in a common situation, the list storing large size xattr is null, so only 4 bytes.

          Show
          hitliuyi Yi Liu added a comment - - edited The max size of the packed byte[] is 32KB * 32 = 1MB... But such big sequential memory allocation could cause full GC when heap is fragmented. We can say this would never happen in real case. Typical the xattr value size is very small (several bytes to tens of bytes, much smaller than 32KB, this is also the design goal of xattr, for system/security namespace xattr used by HDFS core, the size are very small today; if applications use user/trusted xattr to store many large size xattrs, the memory of NN itself will have serious problem, for totally 1MB size xattrs of an INode as in your example, 1TB memory is required if there are only 1MB number Inodes). In real case, even for HDFS transparent encryption which uses xattr to store lots of information (see FileEncryptionInfo), the xattr value size is only 48bytes. Of course the applications may use the xattr wrongly and use it to many store large size (several KB to tens of KB) xattrs, to prevent this, I will add a threshold not to pack those large size xattrs and still store them separately in a list as original. For small size xattr, we will pack them. The threshold is chosen to be 1KB, in a common situation, the list storing large size xattr is null, so only 4 bytes.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 4 new checkstyle issues (total was 517, now 515).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 32s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 9s Pre-build of native portion
          -1 hdfs tests 162m 21s Tests failed in hadoop-hdfs.
              206m 33s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.TestReplaceDatanodeOnFailure



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752018/HDFS-8900.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / feaf034
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12082/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12082/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 4 new checkstyle issues (total was 517, now 515). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 32s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 162m 21s Tests failed in hadoop-hdfs.     206m 33s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.TestReplaceDatanodeOnFailure Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752018/HDFS-8900.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / feaf034 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12082/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12082/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12082/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          I missed to reset the variables at the beginning of XAttrFeature constructor, will update it shortly. (no need, since we create a new XAttrFeature each time).

          Show
          hitliuyi Yi Liu added a comment - - edited I missed to reset the variables at the beginning of XAttrFeature constructor, will update it shortly. (no need, since we create a new XAttrFeature each time).
          Hide
          hitliuyi Yi Liu added a comment - - edited

          The test failures are not related, and findbugs don't exist.
          Hi Andrew Wang, could you give another review for the new updated patch? Thanks.

          Show
          hitliuyi Yi Liu added a comment - - edited The test failures are not related, and findbugs don't exist. Hi Andrew Wang , could you give another review for the new updated patch? Thanks.
          Hide
          hitliuyi Yi Liu added a comment -

          Update patch to fix the checkstyle and add more comment in XAttrFeature.

          Show
          hitliuyi Yi Liu added a comment - Update patch to fix the checkstyle and add more comment in XAttrFeature.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Latest patch should be ok as you added PACK_THRESHOLD.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Latest patch should be ok as you added PACK_THRESHOLD.
          Hide
          andrew.wang Andrew Wang added a comment -

          I like the PACK_THRESHOLD as a solution, doesn't change the core of the patch and I see there's some unittest coverage. I'm +1 still, thanks for working on this Yi and reviewing Brahma.

          Show
          andrew.wang Andrew Wang added a comment - I like the PACK_THRESHOLD as a solution, doesn't change the core of the patch and I see there's some unittest coverage. I'm +1 still, thanks for working on this Yi and reviewing Brahma.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 50s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 1 new checkstyle issues (total was 516, now 511).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 2m 36s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          +1 hdfs tests 163m 27s Tests passed in hadoop-hdfs.
              209m 9s  



          Reason Tests
          FindBugs module:hadoop-hdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752124/HDFS-8900.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3b00eae
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12103/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12103/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 50s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 1 new checkstyle issues (total was 516, now 511). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 36s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion +1 hdfs tests 163m 27s Tests passed in hadoop-hdfs.     209m 9s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752124/HDFS-8900.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3b00eae Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12103/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12103/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Andrew, Colin and Brahma.

          The findbugs doesn't exist. (The existing findbugs were in pre-patch).

          Committed to trunk and branch-2.

          Show
          hitliuyi Yi Liu added a comment - Thanks Andrew, Colin and Brahma. The findbugs doesn't exist. (The existing findbugs were in pre-patch). Committed to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8348 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8348/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8348 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8348/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #309 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/309/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #309 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/309/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1037 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1037/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1037 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1037/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2253/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2253/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #304 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/304/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #304 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/304/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2234 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2234/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2234 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2234/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #296 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/296/)
          HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #296 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/296/ ) HDFS-8900 . Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev eee0d4563c62647cfaaed6605ee713aaf69add78) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
          Hide
          wheat9 Haohui Mai added a comment -

          The findbugs doesn't exist. (The existing findbugs were in pre-patch).

          It appears that this jira introduces two new findbugs warnings in Jenkins.

          https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html

          Show
          wheat9 Haohui Mai added a comment - The findbugs doesn't exist. (The existing findbugs were in pre-patch). It appears that this jira introduces two new findbugs warnings in Jenkins. https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          Hide
          hitliuyi Yi Liu added a comment -

          Yes, Thanks Haohui.
          Seems there is few issue about the Jenkins report, I can't find them in the report of current JIRA, but we do can find them in other JIRA after it's committed.

          Show
          hitliuyi Yi Liu added a comment - Yes, Thanks Haohui. Seems there is few issue about the Jenkins report, I can't find them in the report of current JIRA, but we do can find them in other JIRA after it's committed.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8362/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8362/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/315/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/315/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #320 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/320/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #320 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/320/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1048 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1048/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1048 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1048/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2264 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2264/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2264 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2264/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/307/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/307/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2245/)
          HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2245/ ) HDFS-8963 . Fix incorrect sign extension of xattr length in HDFS-8900 . (Colin Patrick McCabe via yliu) (yliu: rev e166c038c0aaa57b245f985a1c0fadd5fe33c384) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java

            People

            • Assignee:
              hitliuyi Yi Liu
              Reporter:
              hitliuyi Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development