Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12045

Enable LocalFileSystem#setTimes to change atime

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      LocalFileSystem#setTimes method can not change the last access time currently.

      With java.nio.file package in Java 7, we can implement the function easily.

      1. HADOOP-12045.001.patch
        4 kB
        Kazuho Fujii
      2. HADOOP-12045.002.patch
        5 kB
        Kazuho Fujii
      3. HADOOP-12045.003.patch
        5 kB
        Kazuho Fujii
      4. HADOOP-12045.004-1.patch
        3 kB
        Kazuho Fujii
      5. HADOOP-12045.004-2.patch
        8 kB
        Kazuho Fujii
      6. HADOOP-12045.005-1.patch
        9 kB
        Kazuho Fujii
      7. HADOOP-12045.005-2.patch
        14 kB
        Kazuho Fujii
      8. HADOOP-12045.006.patch
        15 kB
        Kazuho Fujii

        Issue Links

          Activity

          Hide
          fjk Kazuho Fujii added a comment -

          I attached a simple patch. The change in DeprecatedRawLocalFileStatus is for testing.

          Show
          fjk Kazuho Fujii added a comment - I attached a simple patch. The change in DeprecatedRawLocalFileStatus is for testing.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 2s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 28s Tests passed in hadoop-common.
              62m 33s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736346/HADOOP-12045.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / eb6bf91
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 2s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 28s Tests passed in hadoop-common.     62m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736346/HADOOP-12045.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / eb6bf91 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6876/console This message was automatically generated.
          Hide
          zxu zhihai xu added a comment -

          Hi Kazuho Fujii, Thanks for filing and working this issue.
          One nit: Could you replace the following code with a function in testSetTimes? I saw these code is repeated multiple times in your patch.

              status = fileSys.getFileStatus(path);
              assertEquals(newModTime, status.getModificationTime());
              assertEquals(newAccTime, status.getAccessTime());
          

          By the way, did you test the patch at window platform?
          cc Chris Nauroth for window platform.

          Show
          zxu zhihai xu added a comment - Hi Kazuho Fujii , Thanks for filing and working this issue. One nit: Could you replace the following code with a function in testSetTimes ? I saw these code is repeated multiple times in your patch. status = fileSys.getFileStatus(path); assertEquals(newModTime, status.getModificationTime()); assertEquals(newAccTime, status.getAccessTime()); By the way, did you test the patch at window platform? cc Chris Nauroth for window platform.
          Hide
          fjk Kazuho Fujii added a comment -

          Hi zhihai xu, thanks for the comment.
          Certainly, the code repeating was not good. I attached a fixed patch.

          I have not test the patch at Windows yet. I don't use classes depending on a platform. So, I think it works. Anyway, we should test it on Windows platform.

          Show
          fjk Kazuho Fujii added a comment - Hi zhihai xu , thanks for the comment. Certainly, the code repeating was not good. I attached a fixed patch. I have not test the patch at Windows yet. I don't use classes depending on a platform. So, I think it works. Anyway, we should test it on Windows platform.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 7s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 40s Tests passed in hadoop-common.
              61m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736814/HADOOP-12045.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a2bd621
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 40s Tests passed in hadoop-common.     61m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736814/HADOOP-12045.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a2bd621 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6893/console This message was automatically generated.
          Hide
          fjk Kazuho Fujii added a comment -

          Excuse me. I made a mistake.

          Show
          fjk Kazuho Fujii added a comment - Excuse me. I made a mistake.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 20s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 5s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 40s Tests passed in hadoop-common.
              62m 39s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736848/HADOOP-12045.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a2bd621
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 5s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 40s Tests passed in hadoop-common.     62m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736848/HADOOP-12045.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a2bd621 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6894/console This message was automatically generated.
          Hide
          zxu zhihai xu added a comment -

          Hi Kazuho Fujii, thanks for updating the patch.
          I don't have window platform so I can't verify it on window platform either.
          See the discussion at HADOOP-11935.
          +1 non-binding if it works on window platform. Thanks again for working on this.

          Show
          zxu zhihai xu added a comment - Hi Kazuho Fujii , thanks for updating the patch. I don't have window platform so I can't verify it on window platform either. See the discussion at HADOOP-11935 . +1 non-binding if it works on window platform. Thanks again for working on this.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Kazuho Fujii. Thank you for the patch. This looks good overall, and I've verified that it's fine on Windows.

          Before we commit this, I'd like to make sure that symlinks are handled correctly. In java.nio.file, symlinks are followed by default, unless the caller passes LinkOption#NOFOLLOW_LINKS. I haven't been able to find documentation that clearly states the contractual behavior for java.io.File.

          I think the important thing for Hadoop is to remain backwards-compatible with current behavior. That means that we'd need to make sure mtime is populated the same way before and after this change when symlinks are present. For atime, the logic should be equivalent.

          A few specific cases I can think of:

          1. The client calls RawLocalFileSystem#getFileLinkStatus, indicating that they want information on the link itself instead of its target.
          2. The argument is a symlink pointing to a directory.
          3. The argument is a symlink pointing to a file.
          4. The argument is a dangling symlink (target does not exist).

          It would be great if we could have test cases for each of these, then apply the RawLocalFileSystem changes, and then verify that thoses tests continue to pass.

          Show
          cnauroth Chris Nauroth added a comment - Hello Kazuho Fujii . Thank you for the patch. This looks good overall, and I've verified that it's fine on Windows. Before we commit this, I'd like to make sure that symlinks are handled correctly. In java.nio.file , symlinks are followed by default, unless the caller passes LinkOption#NOFOLLOW_LINKS . I haven't been able to find documentation that clearly states the contractual behavior for java.io.File . I think the important thing for Hadoop is to remain backwards-compatible with current behavior. That means that we'd need to make sure mtime is populated the same way before and after this change when symlinks are present. For atime, the logic should be equivalent. A few specific cases I can think of: The client calls RawLocalFileSystem#getFileLinkStatus , indicating that they want information on the link itself instead of its target. The argument is a symlink pointing to a directory. The argument is a symlink pointing to a file. The argument is a dangling symlink (target does not exist). It would be great if we could have test cases for each of these, then apply the RawLocalFileSystem changes, and then verify that thoses tests continue to pass.
          Hide
          fjk Kazuho Fujii added a comment -

          Hi, Chris Nauroth. Thanks for a kind comment.

          I attached two more patches along your advise:

          . The former includes additional tests for the current setTimes method;
          the later includes the tests and the new setTimes method. We can confirm the backword-compatibility with the them.

          Show
          fjk Kazuho Fujii added a comment - Hi, Chris Nauroth . Thanks for a kind comment. I attached two more patches along your advise: HADOOP-12045 .004-1.patch HADOOP-12045 .004-2.patch . The former includes additional tests for the current setTimes method; the later includes the tests and the new setTimes method. We can confirm the backword-compatibility with the them.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 6s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 9s Tests passed in hadoop-common.
              61m 30s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740287/HADOOP-12045.004-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 295d678
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 6s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 9s Tests passed in hadoop-common.     61m 30s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740287/HADOOP-12045.004-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 295d678 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6988/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Kazuho Fujii, thank you for the updated patch. I have just 2 more small comments.

          1. Nitpick: In testSetTimesSymlinkToDir, please use "dir" instead of "file" for the sub-directory name.
          2. I think we also can update TestCopyPreserveFlag as part of this patch. Currently, it only checks mtime, but now it can check atime too.

          After addressing that, I think this will be all done.

          Show
          cnauroth Chris Nauroth added a comment - Kazuho Fujii , thank you for the updated patch. I have just 2 more small comments. Nitpick: In testSetTimesSymlinkToDir , please use "dir" instead of "file" for the sub-directory name. I think we also can update TestCopyPreserveFlag as part of this patch. Currently, it only checks mtime, but now it can check atime too. After addressing that, I think this will be all done.
          Hide
          fjk Kazuho Fujii added a comment -

          Hi Chris Nauroth. Thank you for the comment.

          1. Nitpick: In testSetTimesSymlinkToDir, please use "dir" instead of "file" for the sub-directory name.

          I fixed it. Thanks for pointing it out.

          2. I think we also can update TestCopyPreserveFlag as part of this patch. Currently, it only checks mtime, but now it can check atime too.

          I updated it. I refactored the code at the same time. Is this acceptable? The just refactored version is included in HADOOP-12045.005-1.patch.

          Show
          fjk Kazuho Fujii added a comment - Hi Chris Nauroth . Thank you for the comment. 1. Nitpick: In testSetTimesSymlinkToDir, please use "dir" instead of "file" for the sub-directory name. I fixed it. Thanks for pointing it out. 2. I think we also can update TestCopyPreserveFlag as part of this patch. Currently, it only checks mtime, but now it can check atime too. I updated it. I refactored the code at the same time. Is this acceptable? The just refactored version is included in HADOOP-12045 .005-1.patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 46s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 8s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 36s Tests passed in hadoop-common.
              62m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740864/HADOOP-12045.005-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 63d40d5
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 8s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 36s Tests passed in hadoop-common.     62m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740864/HADOOP-12045.005-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 63d40d5 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6998/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Kazuho Fujii, thank you for the update.

          Unfortunately, the new symlink tests do not pass on Windows. This is a consequence of the way symlink integration works on Windows right now. On Windows, a call to RawLocalFileSystem#getFileLinkStatus returns a FileStatus populated with the symlink as the path, but the other attributes are populated from the symlink target. This breaks the assertions that calling setTimes on the link doesn't alter the times reported by subsequent getFileLinkStatus calls.

          This is a known limitation unrelated to the current patch, and we already have comments marking TODO's around native stat support for Windows. I think it's appropriate to skip these tests on Windows for now. You can do that by adding overrides in TestSymlinkLocalFS for each of the new test methods added to SymlinkBaseTest. The override just checks if it's running on Windows, and then delegates up to the superclass to run the test. Here is an existing example:

            @Override
            public void testCreateDanglingLink() throws IOException {
              // Dangling symlinks are not supported on Windows local file system.
              assumeTrue(!Path.WINDOWS);
              super.testCreateDanglingLink();
            }
          

          After that's done, I suspect it will be the final version of the patch. Thanks for sticking with this!

          Show
          cnauroth Chris Nauroth added a comment - Kazuho Fujii , thank you for the update. Unfortunately, the new symlink tests do not pass on Windows. This is a consequence of the way symlink integration works on Windows right now. On Windows, a call to RawLocalFileSystem#getFileLinkStatus returns a FileStatus populated with the symlink as the path, but the other attributes are populated from the symlink target. This breaks the assertions that calling setTimes on the link doesn't alter the times reported by subsequent getFileLinkStatus calls. This is a known limitation unrelated to the current patch, and we already have comments marking TODO's around native stat support for Windows. I think it's appropriate to skip these tests on Windows for now. You can do that by adding overrides in TestSymlinkLocalFS for each of the new test methods added to SymlinkBaseTest . The override just checks if it's running on Windows, and then delegates up to the superclass to run the test. Here is an existing example: @Override public void testCreateDanglingLink() throws IOException { // Dangling symlinks are not supported on Windows local file system. assumeTrue(!Path.WINDOWS); super .testCreateDanglingLink(); } After that's done, I suspect it will be the final version of the patch. Thanks for sticking with this!
          Hide
          fjk Kazuho Fujii added a comment -

          Chris Nauroth, thanks for testing the patch.

          I updated the patch again. I override new test methods and check if runnnig on Windows. Is it OK?

          Thank you for repeatedly helping.

          Show
          fjk Kazuho Fujii added a comment - Chris Nauroth , thanks for testing the patch. I updated the patch again. I override new test methods and check if runnnig on Windows. Is it OK? Thank you for repeatedly helping.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 21s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 4s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 2s Tests passed in hadoop-common.
              61m 6s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12743586/HADOOP-12045.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 2eae130
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 2s Tests passed in hadoop-common.     61m 6s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743586/HADOOP-12045.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2eae130 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7144/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v06. I confirmed that this version passes fine on Windows. Kazuho, thank you for contributing the patch!

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v06. I confirmed that this version passes fine on Windows. Kazuho, thank you for contributing the patch!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8117 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8117/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8117 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8117/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #979 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/979/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #979 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/979/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #249 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/249/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #249 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/249/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2176/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2176/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/237/)
          HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/237/ ) HADOOP-12045 . Enable LocalFileSystem#setTimes to change atime. Contributed by Kazuho Fujii. (cnauroth: rev ed1e3ce482f679ae2fad43a203f6578d7af59327) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Kazuho Fujii. FYI, there is a follow-up bug fix to this being discussed on HADOOP-12258.

          Show
          cnauroth Chris Nauroth added a comment - Hi Kazuho Fujii . FYI, there is a follow-up bug fix to this being discussed on HADOOP-12258 .

            People

            • Assignee:
              fjk Kazuho Fujii
              Reporter:
              fjk Kazuho Fujii
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development