Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:

      Description

      We have two special paths within /.reserved now, /.reserved/.inodes and /.reserved/raw. It seems like we should be able to list /.reserved to see them.

      1. HDFS-7087.001.patch
        20 kB
        Xiao Chen
      2. HDFS-7087.002.patch
        23 kB
        Xiao Chen
      3. HDFS-7087.003.branch-2.patch
        24 kB
        Yi Liu
      4. HDFS-7087.003.patch
        24 kB
        Xiao Chen
      5. HDFS-7087.draft.patch
        21 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Andrew, we can add an linux-like option "-a" to list hidden files, does that sound good to you? I can work out a solution here. Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Andrew, we can add an linux-like option "-a" to list hidden files, does that sound good to you? I can work out a solution here. Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Yongjun, thanks for taking a look at this. AFAIK we don't need a "-a" option, since the HDFS shell doesn't hide paths starting with a "." like UNIXy shells. The core issue here is that right now, FSDirectory#resolvePath doesn't resolve "/.reserved" to anything, so we get FileNotFound when trying to do any operations on it. Reserved paths are all very special cased right now.

          I think what we need to do is create two fake FileStatuses for when people list "/.reserved", and return those. It'd also be nice for getFileInfo on /.reserved directly to work. I think we shouldn't show .reserved when listing / though, since that could lead to bad things if someone ran e.g. a distcp on /.

          Show
          andrew.wang Andrew Wang added a comment - Hi Yongjun, thanks for taking a look at this. AFAIK we don't need a "-a" option, since the HDFS shell doesn't hide paths starting with a "." like UNIXy shells. The core issue here is that right now, FSDirectory#resolvePath doesn't resolve "/.reserved" to anything, so we get FileNotFound when trying to do any operations on it. Reserved paths are all very special cased right now. I think what we need to do is create two fake FileStatuses for when people list "/.reserved", and return those. It'd also be nice for getFileInfo on /.reserved directly to work. I think we shouldn't show .reserved when listing / though, since that could lead to bad things if someone ran e.g. a distcp on / .
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Andrew, thanks for the explanation, and sorry for late reply. Based on your explanation, my understanding is, given [path /a/b/.reserved, command hadoop fs -ls /a/b/ should not list /a/b/.reserved, however, command hadoop fs -ls /a/b/.reserved should list it, and hadoop fs -lsr /a/b/.reserved should list everything under .reserved? Maybe you can add a complete list of this kind of scenarios to the jira description? thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Andrew, thanks for the explanation, and sorry for late reply. Based on your explanation, my understanding is, given [path /a/b/.reserved , command hadoop fs -ls /a/b/ should not list /a/b/.reserved , however, command hadoop fs -ls /a/b/.reserved should list it, and hadoop fs -lsr /a/b/.reserved should list everything under .reserved ? Maybe you can add a complete list of this kind of scenarios to the jira description? thanks.
          Hide
          clamb Charles Lamb added a comment -

          Hi Yongjun,

          A slight clarification:

          .reserved is only top level, so you can't have /a/b/.reserved (well, you could, but it wouldn't be the 'special' .reserved). You only need to worry about /.reserved. So the only thing this Jira is concerned with is

          ls /.reserved

          which should show two items: raw and .inodes

          Show
          clamb Charles Lamb added a comment - Hi Yongjun, A slight clarification: .reserved is only top level, so you can't have /a/b/.reserved (well, you could, but it wouldn't be the 'special' .reserved). You only need to worry about /.reserved. So the only thing this Jira is concerned with is ls /.reserved which should show two items: raw and .inodes
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Charles, thanks for the clarification, which helped me to realize that it's exactly what Andrew stated in the jira description.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Charles, thanks for the clarification, which helped me to realize that it's exactly what Andrew stated in the jira description .
          Hide
          sureshms Suresh Srinivas added a comment -

          We have two special paths within /.reserved now, /.reserved/.inodes and /.reserved/raw. It seems like we should be able to list /.reserved to see them.

          Andrew Wang, this will be very useful. Good idea!

          Show
          sureshms Suresh Srinivas added a comment - We have two special paths within /.reserved now, /.reserved/.inodes and /.reserved/raw. It seems like we should be able to list /.reserved to see them. Andrew Wang , this will be very useful. Good idea!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Moving features/enhancements out of previously closed releases into the next minor release 2.8.0.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Moving features/enhancements out of previously closed releases into the next minor release 2.8.0.
          Hide
          xiaochen Xiao Chen added a comment -

          Hi Andrew Wang,

          Thanks for reporting the issue and further explain it in the comment.
          I'm working on this JIRA, and have a few points I want to clarify.

          When listing /.reserved, we will be able to see hard-coded FileStatuses of /.reserved/.inodes and /.reserved/raw:
          1. Is this visible to all users? Or just the superuser? Does it make sense if I set the permission as 755?
          2. What atime and mtime would be preferred? I know they aren't supposed to be modified. But I'm not sure when ls it, what timestamp would be meaningful for them.

          Show
          xiaochen Xiao Chen added a comment - Hi Andrew Wang , Thanks for reporting the issue and further explain it in the comment. I'm working on this JIRA, and have a few points I want to clarify. When listing /.reserved , we will be able to see hard-coded FileStatuses of /.reserved/.inodes and /.reserved/raw : 1. Is this visible to all users? Or just the superuser? Does it make sense if I set the permission as 755? 2. What atime and mtime would be preferred? I know they aren't supposed to be modified. But I'm not sure when ls it, what timestamp would be meaningful for them.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Xiao, thanks for your interest in this one!

          • I think it's fine if it's visible to all users, since access to actual file information is still controlled
          • /.reserved/raw is for admins only right now, so a 770 permission owned by the superuser/supergroup would be appropriate.
          • I'm not sure about the permissions for /.reserved/.inodes. You can't create a file by inode, but might be able to delete. Will let you poke around and determine the right thing here.
          • atime and mtime, perhas we set it to when the cluster was initially formatted? I don't see much value in making it more up-to-date than that.
          Show
          andrew.wang Andrew Wang added a comment - Hi Xiao, thanks for your interest in this one! I think it's fine if it's visible to all users, since access to actual file information is still controlled /.reserved/raw is for admins only right now, so a 770 permission owned by the superuser/supergroup would be appropriate. I'm not sure about the permissions for /.reserved/.inodes . You can't create a file by inode, but might be able to delete. Will let you poke around and determine the right thing here. atime and mtime, perhas we set it to when the cluster was initially formatted? I don't see much value in making it more up-to-date than that.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks very much Andrew Wang for the explanation!

          I have created a draft patch to support listing /.reserved. Would you please review and see if I'm on the right path? BTW There is a dependency on HDFS-8164 to get the format time, so if you test this patch the mtime is currently 0. I'll get HDFS-8164 done asap.

          Brief description of the draft:

          • Add fake statuses of /.reserved, /.reserved/raw, /.reserved/.inodes. The first is hard-coded, the latter 2 are lazy-initialized to read cTime, which has to happen strong*after*strong FSDirectory's initialization (specifically, after FSNamesystem loads fsimage.
          • FSDirStatAndListingOp#getFileInfo has to work so that Globber#getFileStatus can return non-null status in order for the shell to proceed to the actual command (fs.listStatus)
          • FSDirStatAndListingOp#getListing returns the statuses.
          • FSDirectory#resolvePath doesn't seem to require any change - no special resolution needed.
          • Guard all FSDir**Ops from /.reserved path.
          • Test cases

          Any feedback is appreciated!

          Show
          xiaochen Xiao Chen added a comment - Thanks very much Andrew Wang for the explanation! I have created a draft patch to support listing /.reserved. Would you please review and see if I'm on the right path? BTW There is a dependency on HDFS-8164 to get the format time, so if you test this patch the mtime is currently 0. I'll get HDFS-8164 done asap. Brief description of the draft: Add fake statuses of /.reserved, /.reserved/raw, /.reserved/.inodes. The first is hard-coded, the latter 2 are lazy-initialized to read cTime, which has to happen strong*after*strong FSDirectory 's initialization (specifically, after FSNamesystem loads fsimage. FSDirStatAndListingOp#getFileInfo has to work so that Globber#getFileStatus can return non-null status in order for the shell to proceed to the actual command ( fs.listStatus ) FSDirStatAndListingOp#getListing returns the statuses. FSDirectory#resolvePath doesn't seem to require any change - no special resolution needed. Guard all FSDir**Ops from /.reserved path. Test cases Any feedback is appreciated!
          Hide
          xiaochen Xiao Chen added a comment -

          Now that HDFS-8164 is resolved (thanks Yongjun Zhang!), I'm posting patch 001 for review.
          Beside the above description, I think I should also mention that the patch just supports listing /.reserved while keeping other behaviors untouched, so operations on /.reserved/raw and /.reserved/.inodes (or their subdirs) behaves the same as before.
          Any comments are highly appreciated. Thank you in advance.

          Show
          xiaochen Xiao Chen added a comment - Now that HDFS-8164 is resolved (thanks Yongjun Zhang !), I'm posting patch 001 for review. Beside the above description, I think I should also mention that the patch just supports listing /.reserved while keeping other behaviors untouched, so operations on /.reserved/raw and /.reserved/.inodes (or their subdirs) behaves the same as before. Any comments are highly appreciated. Thank you in advance.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Xiao, thanks for working on this! Some review comments:

          • Would appreciate javadoc and comments explaining the new methods, and also rationale for the new fake FileStatuses.
          • Any reason you set the sticky bit?
          • The isExactReservedName checks, noticed there's no check for rename. This seems brittle in general since we have to make sure it's present in everything besides list. Is the intent of these checks to provide consistent behavior? Do we think this is necessary, or can we simply rely on the existing behavior?
          • This enables listing /.reserved/raw/.reserved/, which is a little weird.
          • There are some acrobatics to get the cTime. Is this something we could set during FSImage loading? Would let us avoid the null check.
          Show
          andrew.wang Andrew Wang added a comment - Hi Xiao, thanks for working on this! Some review comments: Would appreciate javadoc and comments explaining the new methods, and also rationale for the new fake FileStatuses. Any reason you set the sticky bit? The isExactReservedName checks, noticed there's no check for rename. This seems brittle in general since we have to make sure it's present in everything besides list. Is the intent of these checks to provide consistent behavior? Do we think this is necessary, or can we simply rely on the existing behavior? This enables listing /.reserved/raw/.reserved/, which is a little weird. There are some acrobatics to get the cTime. Is this something we could set during FSImage loading? Would let us avoid the null check.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 47s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 43s There were no new javac warning messages.
          +1 javadoc 12m 1s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 122, now 126).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 46s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 33s Pre-build of native portion
          -1 hdfs tests 188m 3s Tests failed in hadoop-hdfs.
              239m 25s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestReplaceDatanodeOnFailure



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765743/HDFS-7087.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4f6e842
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12898/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12898/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 43s There were no new javac warning messages. +1 javadoc 12m 1s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 122, now 126). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 33s Pre-build of native portion -1 hdfs tests 188m 3s Tests failed in hadoop-hdfs.     239m 25s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestReplaceDatanodeOnFailure Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765743/HDFS-7087.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4f6e842 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12898/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12898/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12898/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot Andrew for the review and comments! I have attached patch 002 to address your comments.

          Would appreciate javadoc and comments explaining the new methods, and also rationale for the new fake FileStatuses.

          Added to FSDirectory#createReservedStatuses, please see if they make sense to you.

          Any reason you set the sticky bit?

          I initially thought we should set it to indicate the dir is not intended for unprivileged deletion. On a second thought, this is not needed and confusing, so I removed it.

          The isExactReservedName checks, noticed there's no check for rename. This seems brittle in general since we have to make sure it's present in everything besides list. Is the intent of these checks to provide consistent behavior? Do we think this is necessary, or can we simply rely on the existing behavior?

          I agree that it's fragile because we have to protect everything besides list. But we can’t rely on the existing behavior, because currently it’s protected by getFileInfo while parsing from the shell - before this patch, getFileInfo returns null for /.reserved so that the command isn't get processed and hence never reach to FSxxxxOp (except mkdirs). Now that we have to enable getFileInfo in order for ls to work, I think we have to protect other commands by ourselves. Below is a sample stack trace of the getFileInfo call.

          	 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:80)
          	 at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source:-1)
          	 at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1667)
          	 at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1269)
          	 at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1265)
          	 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
          	 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1265)
          	 at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:62)
          	 at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
          	 at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
          	 at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1670)
          	 at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
          	 at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239)
          	 at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222)
          	 at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
          	 at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
          	 at org.apache.hadoop.fs.FsShell.run(FsShell.java:310)
          	 at org.apache.hadoop.hdfs.TestDFSShell.runCmd(TestDFSShell.java:971)
          

          This enables listing /.reserved/raw/.reserved/, which is a little weird.

          Agree and good catch! Fixed.

          There are some acrobatics to get the cTime. Is this something we could set during FSImage loading? Would let us avoid the null check.

          Good idea, updated.

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot Andrew for the review and comments! I have attached patch 002 to address your comments. Would appreciate javadoc and comments explaining the new methods, and also rationale for the new fake FileStatuses. Added to FSDirectory#createReservedStatuses , please see if they make sense to you. Any reason you set the sticky bit? I initially thought we should set it to indicate the dir is not intended for unprivileged deletion. On a second thought, this is not needed and confusing, so I removed it. The isExactReservedName checks, noticed there's no check for rename. This seems brittle in general since we have to make sure it's present in everything besides list. Is the intent of these checks to provide consistent behavior? Do we think this is necessary, or can we simply rely on the existing behavior? I agree that it's fragile because we have to protect everything besides list. But we can’t rely on the existing behavior, because currently it’s protected by getFileInfo while parsing from the shell - before this patch, getFileInfo returns null for /.reserved so that the command isn't get processed and hence never reach to FSxxxxOp (except mkdirs). Now that we have to enable getFileInfo in order for ls to work, I think we have to protect other commands by ourselves. Below is a sample stack trace of the getFileInfo call. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:80) at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source:-1) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1667) at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1269) at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1265) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1265) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:62) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1670) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:166) at org.apache.hadoop.fs.FsShell.run(FsShell.java:310) at org.apache.hadoop.hdfs.TestDFSShell.runCmd(TestDFSShell.java:971) This enables listing /.reserved/raw/.reserved/, which is a little weird. Agree and good catch! Fixed. There are some acrobatics to get the cTime. Is this something we could set during FSImage loading? Would let us avoid the null check. Good idea, updated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766177/HDFS-7087.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8175c4f
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13061/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766177/HDFS-7087.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8175c4f Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13061/console This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Looks good, thanks for revving Xiao. Patch needs a small rebase for TestDFSShell. Only two little nits:

          • FSNamesystem.getCTime is marked VisibleForTesting. What we could do is pass the cTime as a parameter to createReservedStatuses which decouples FSN and FSDir a bit more. Also would be good to add something to createReservedStatuses javadoc about how this is called only once during NN initialization.
          • Add a Precondition check that reservedStatuses is not null before use? Maybe in getReservedStatuses. This also serves as documentation that createReservedStatuses should be called first.
          Show
          andrew.wang Andrew Wang added a comment - Looks good, thanks for revving Xiao. Patch needs a small rebase for TestDFSShell. Only two little nits: FSNamesystem.getCTime is marked VisibleForTesting. What we could do is pass the cTime as a parameter to createReservedStatuses which decouples FSN and FSDir a bit more. Also would be good to add something to createReservedStatuses javadoc about how this is called only once during NN initialization. Add a Precondition check that reservedStatuses is not null before use? Maybe in getReservedStatuses. This also serves as documentation that createReservedStatuses should be called first.
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you for the additional comments Andrew!
          Attached patch003 is rebased to latest trunk and addresses all of your comments.

          Show
          xiaochen Xiao Chen added a comment - Thank you for the additional comments Andrew! Attached patch003 is rebased to latest trunk and addresses all of your comments.
          Hide
          andrew.wang Andrew Wang added a comment -

          I just started a new build on Jenkins, not sure why precommit hasn't picked it up.

          Show
          andrew.wang Andrew Wang added a comment - I just started a new build on Jenkins, not sure why precommit hasn't picked it up.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 19m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 9m 28s There were no new javac warning messages.
          +1 javadoc 12m 14s There were no new javadoc warning messages.
          +1 release audit 0m 29s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 50s The applied patch generated 4 new checkstyle issues (total was 385, now 388).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 54s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 3m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 51s Pre-build of native portion
          -1 hdfs tests 54m 19s Tests failed in hadoop-hdfs.
              107m 56s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestReplaceDatanodeOnFailure



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767709/HDFS-7087.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a24c6e8
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13117/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13117/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 9m 28s There were no new javac warning messages. +1 javadoc 12m 14s There were no new javadoc warning messages. +1 release audit 0m 29s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 50s The applied patch generated 4 new checkstyle issues (total was 385, now 388). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 54s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 3m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 51s Pre-build of native portion -1 hdfs tests 54m 19s Tests failed in hadoop-hdfs.     107m 56s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestReplaceDatanodeOnFailure Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767709/HDFS-7087.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a24c6e8 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13117/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13117/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13117/console This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM I'll fix the little whitespace issues at commit time

          Show
          andrew.wang Andrew Wang added a comment - LGTM I'll fix the little whitespace issues at commit time
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks for working on this Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk and branch-2, thanks for working on this Xiao!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8685 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8685/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8685 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8685/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1302 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1302/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1302 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1302/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          Hide
          jlowe Jason Lowe added a comment -

          Looks like this broke the branch-2 build:

          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[126,7] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types;
            required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte
            found: int,boolean,int,int,int,int,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,<nulltype>,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype>
            reason: actual and formal argument lists differ in length
          [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[350,29] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types;
            required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte
            found: int,boolean,int,int,long,long,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,java.lang.String,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype>
            reason: actual and formal argument lists differ in length
          [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[354,26] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types;
            required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte
            found: int,boolean,int,int,long,long,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,java.lang.String,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype>
            reason: actual and formal argument lists differ in length
          [INFO] 3 errors 
          
          Show
          jlowe Jason Lowe added a comment - Looks like this broke the branch-2 build: [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[126,7] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types; required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte found: int,boolean,int,int,int,int,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,<nulltype>,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype> reason: actual and formal argument lists differ in length [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[350,29] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types; required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte found: int,boolean,int,int,long,long,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,java.lang.String,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype> reason: actual and formal argument lists differ in length [ERROR] /hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:[354,26] constructor HdfsFileStatus in class org.apache.hadoop.hdfs.protocol.HdfsFileStatus cannot be applied to given types; required: long,boolean,int,long,long,long,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,java.lang.String,byte[],byte[],long,int,org.apache.hadoop.fs.FileEncryptionInfo,byte found: int,boolean,int,int,long,long,org.apache.hadoop.fs.permission.FsPermission,<nulltype>,java.lang.String,<nulltype>,byte[],long,int,<nulltype>,byte,<nulltype> reason: actual and formal argument lists differ in length [INFO] 3 errors
          Hide
          hitliuyi Yi Liu added a comment -

          That's right. I will help to revert it from branch-2 and re-commit it shortly.

          Show
          hitliuyi Yi Liu added a comment - That's right. I will help to revert it from branch-2 and re-commit it shortly.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #581 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/581/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #581 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/581/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Hide
          hitliuyi Yi Liu added a comment -

          recommit to branch-2, and attach the patch of branch-2 for track.

          Show
          hitliuyi Yi Liu added a comment - recommit to branch-2, and attach the patch of branch-2 for track.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2513 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2513/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2513 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2513/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/566/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/566/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #524 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/524/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #524 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/524/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Jason for reporting the error, and thanks Yi for fixing it!

          Show
          xiaochen Xiao Chen added a comment - Thanks Jason for reporting the error, and thanks Yi for fixing it!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2461 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2461/)
          HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2461 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2461/ ) HDFS-7087 . Ability to list /.reserved. Contributed by Xiao Chen. (wang: rev 3dadf369d550c2ae393b751cb5a184dbfe2814df) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Yea, thanks for hopping on this Jason and Yi, the cherry-pick was clean but that doesn't always mean no compile error.

          Show
          andrew.wang Andrew Wang added a comment - Yea, thanks for hopping on this Jason and Yi, the cherry-pick was clean but that doesn't always mean no compile error.
          Hide
          hitliuyi Yi Liu added a comment -

          I think there is no problem, thanks Andrew, the cherry-pick shows no conflicts, I think we usually don't check the compiling before committing if there is no patch conflicts. Then we fix it if we find compile error later.

          Thanks Xiao Chen and Jason.

          Show
          hitliuyi Yi Liu added a comment - I think there is no problem, thanks Andrew, the cherry-pick shows no conflicts, I think we usually don't check the compiling before committing if there is no patch conflicts. Then we fix it if we find compile error later. Thanks Xiao Chen and Jason.
          Hide
          zhz Zhe Zhang added a comment -

          Quick note that I'm working on backporting HDFS-10744 to branch-2.7; since this change is not in branch-2.7, the overlapping part won't be backported.

          If someone plans to backport this to branch-2.7, please remember to add the optimization from HDFS-10744.

          Show
          zhz Zhe Zhang added a comment - Quick note that I'm working on backporting HDFS-10744 to branch-2.7; since this change is not in branch-2.7, the overlapping part won't be backported. If someone plans to backport this to branch-2.7, please remember to add the optimization from HDFS-10744 .

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development