Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      S3A has been made accessible through the FileContext API.

      Description

      Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.

      1. HADOOP-11262.patch
        3 kB
        Thomas Demoor
      2. HADOOP-11262-10.patch
        32 kB
        Pieter Reuse
      3. HADOOP-11262-2.patch
        6 kB
        Thomas Demoor
      4. HADOOP-11262-3.patch
        7 kB
        Thomas Demoor
      5. HADOOP-11262-4.patch
        8 kB
        Thomas Demoor
      6. HADOOP-11262-5.patch
        9 kB
        Thomas Demoor
      7. HADOOP-11262-6.patch
        32 kB
        Pieter Reuse
      8. HADOOP-11262-7.patch
        31 kB
        Pieter Reuse
      9. HADOOP-11262-8.patch
        32 kB
        Pieter Reuse
      10. HADOOP-11262-9.patch
        32 kB
        Pieter Reuse

        Issue Links

          Activity

          Hide
          thodemoor Thomas Demoor added a comment -

          Change proposed by Takenori Sato in the comments of HADOOP-10400 .

          Mimics hadoop-azure's Wasb.java

          Show
          thodemoor Thomas Demoor added a comment - Change proposed by Takenori Sato in the comments of HADOOP-10400 . Mimics hadoop-azure's Wasb.java
          Hide
          thodemoor Thomas Demoor added a comment -

          Added 2 small test just to assure that S3A can be used from a FileContext.

          To enable S3A your configuration needs to contain:

            <property>
              <name>fs.AbstractFileSystem.s3a.impl</name>
              <value>org.apache.hadoop.fs.s3a.S3A</value>
            </property>
          

          For the tests this is done automatically.

          Show
          thodemoor Thomas Demoor added a comment - Added 2 small test just to assure that S3A can be used from a FileContext. To enable S3A your configuration needs to contain: <property> <name>fs.AbstractFileSystem.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3A</value> </property> For the tests this is done automatically.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679839/HADOOP-11262-2.patch
          against trunk revision 80d7d18.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5043//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5043//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679839/HADOOP-11262-2.patch against trunk revision 80d7d18. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5043//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5043//console This message is automatically generated.
          Hide
          thodemoor Thomas Demoor added a comment -

          Refactored tests to use the (recently introduced) S3ATestUtils.

          Show
          thodemoor Thomas Demoor added a comment - Refactored tests to use the (recently introduced) S3ATestUtils.
          Hide
          thodemoor Thomas Demoor added a comment -

          new patch uploaded

          Show
          thodemoor Thomas Demoor added a comment - new patch uploaded
          Hide
          stevel@apache.org Steve Loughran added a comment -

          TestS3A

          1. replace .* import with specific class imports
          2. you shouldn't have to set the s3a.impl file; it should instead go into core-default.xml
          3. testS3AStatus(): assertTrue/false should include an error message, ideally with the errant values
          4. testS3ACreateFileInSubDir should close FS in a try/finally block. You can use the Java7 construct to do this now.
          Show
          stevel@apache.org Steve Loughran added a comment - TestS3A replace .* import with specific class imports you shouldn't have to set the s3a.impl file; it should instead go into core-default.xml testS3AStatus() : assertTrue/false should include an error message, ideally with the errant values testS3ACreateFileInSubDir should close FS in a try/finally block. You can use the Java7 construct to do this now.
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 4 addresses Steve Loughran's remarks.

          Show
          thodemoor Thomas Demoor added a comment - Patch 4 addresses Steve Loughran 's remarks.
          Hide
          cnauroth Chris Nauroth added a comment -

          you shouldn't have to set the s3a.impl file; it should instead go into core-default.xml

          I've been thinking that we really ought to have ServiceLoader support for AbstractFileSystem, so that S3A and others can register themselves in their jar manifest instead of the configuration files, just like FileSystem. I filed HADOOP-11527 for this.

          No need to hold up the work happening here waiting for it though. This is just FYI.

          Show
          cnauroth Chris Nauroth added a comment - you shouldn't have to set the s3a.impl file; it should instead go into core-default.xml I've been thinking that we really ought to have ServiceLoader support for AbstractFileSystem , so that S3A and others can register themselves in their jar manifest instead of the configuration files, just like FileSystem . I filed HADOOP-11527 for this. No need to hold up the work happening here waiting for it though. This is just FYI.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693262/HADOOP-11262-4.patch
          against trunk revision a761bf8.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693262/HADOOP-11262-4.patch against trunk revision a761bf8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//console This message is automatically generated.
          Hide
          thodemoor Thomas Demoor added a comment -

          patch 5: Adds property to the site documentation and updates description in core-default

          Show
          thodemoor Thomas Demoor added a comment - patch 5: Adds property to the site documentation and updates description in core-default
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699272/HADOOP-11262-5.patch
          against trunk revision f24a567.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699272/HADOOP-11262-5.patch against trunk revision f24a567. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1.

          Test run completed.

          But: not enough new tests for what is a major addition to S3a. There are some extra tests in hadoop-common-test which can be subclassed to help qualify FileContext implementations.

          At a glance, I can see the following

          FileContextCreateMkdirBaseTest
          FileContextMainOperationsBaseTest
          FileContextUtilBase
          FileContextURIBase
          
          Show
          stevel@apache.org Steve Loughran added a comment - -1. Test run completed. But: not enough new tests for what is a major addition to S3a. There are some extra tests in hadoop-common-test which can be subclassed to help qualify FileContext implementations. At a glance, I can see the following FileContextCreateMkdirBaseTest FileContextMainOperationsBaseTest FileContextUtilBase FileContextURIBase
          Hide
          thodemoor Thomas Demoor added a comment -

          Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere.

          Show
          thodemoor Thomas Demoor added a comment - Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere.
          Hide
          cnauroth Chris Nauroth added a comment -

          Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere.

          This is a gap that needs to be addressed in hadoop-azure too, but we can track that in a separate jira.

          Show
          cnauroth Chris Nauroth added a comment - Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere. This is a gap that needs to be addressed in hadoop-azure too, but we can track that in a separate jira.
          Hide
          jghoman Jakob Homan added a comment -

          So, to be clear, Steve Loughran, you're not vetoing the approach, just the patch until more tests are added? Thomas Demoor, are you interested in adding more tests?

          Show
          jghoman Jakob Homan added a comment - So, to be clear, Steve Loughran , you're not vetoing the approach, just the patch until more tests are added? Thomas Demoor , are you interested in adding more tests?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          that's right: nothing wrong with the core code, we just need the tests to show they work. Ideally even spinning up a MiniYarnCluster using S3a as the FS

          Show
          stevel@apache.org Steve Loughran added a comment - that's right: nothing wrong with the core code, we just need the tests to show they work. Ideally even spinning up a MiniYarnCluster using S3a as the FS
          Hide
          thodemoor Thomas Demoor added a comment -

          Jakob Homan,
          we're indeed still planning to add these tests. You'll see some progress here from me or Pieter Reuse, who recently joined my team, soon (I want to wrap up some work on HADOOP-9565 first).

          Show
          thodemoor Thomas Demoor added a comment - Jakob Homan , we're indeed still planning to add these tests. You'll see some progress here from me or Pieter Reuse , who recently joined my team, soon (I want to wrap up some work on HADOOP-9565 first).
          Hide
          thodemoor Thomas Demoor added a comment -

          We (well, Pieter Reuse actually ) have implemented all FileContext tests + a test that spins up MiniYARNCluster. Some minor code additions / changes to make things work. Have been running the patch on our test cluster for a week now. So far things look good but want to check some more corner cases before uploading the patch.

          Show
          thodemoor Thomas Demoor added a comment - We (well, Pieter Reuse actually ) have implemented all FileContext tests + a test that spins up MiniYARNCluster. Some minor code additions / changes to make things work. Have been running the patch on our test cluster for a week now. So far things look good but want to check some more corner cases before uploading the patch.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Patch 6:

          As requested, expanded patch 5 with tests by extending the following tests, overriding them with S3A specifics:

          • TestFileContext.java
          • FileContextCreateMkdirBaseTest.java
          • FileContextMainOperationsBaseTest.java
          • FCStatisticsBaseTest.java
          • FileContextURIBase.java
          • FileContextUtilBase.java

          In doing so, fixed following bugs in FileContextMainOperationsBaseTest.java:

          • line 1169: creating a symlink on an FS that doesn't support this throws an UnsupportedOperationException, not an IOException (see FileContext.java:1441).
          • lines 1252 and 1313: the contract of read() is not to read the whole file - that's the contract of readFully(). For this reason tests assuming that the whole file has been read should use readFully() instead of read().

          And added an enhancement for object storage systems in the same file:

          • line 1238: an object storage system throws an IOException as a file does not exist before the file is closed (nor does it have a checksum at that moment). This object-storage issue is resolved by changing the order of fc.setVerifyChecksum(true, path) and out.write(data, 0, data.length), while this does not impact the behaviour on hdfs or other file systems.

          Discovered and patched the following related bugs in S3A:

          • Bugfix in S3AFileSystem.java: ports on s3 should be ignored, which corresponds with a value of -1 (instead of the default 0 in FileSystem).
          • Another bugfix is in S3AFileStatus.java: getModificationTime() is overwritten for directories. It returns System.currentTimeMillis() because an ObjectStore does not keep track of modification-times of directories. Because some parts of the Hadoop ecosystem use modification time to ignore or delete "old" directories (e.g. the YarnHistorySever), returning 0 for directories is not the best option here.

          Added TestS3AMiniYarnCluster.java, which runs a simple WordCount-MapReduce job on a YarnMiniCluster using S3A as filesystem.

          Show
          PieterReuse Pieter Reuse added a comment - Patch 6: As requested, expanded patch 5 with tests by extending the following tests, overriding them with S3A specifics: TestFileContext.java FileContextCreateMkdirBaseTest.java FileContextMainOperationsBaseTest.java FCStatisticsBaseTest.java FileContextURIBase.java FileContextUtilBase.java In doing so, fixed following bugs in FileContextMainOperationsBaseTest.java : line 1169 : creating a symlink on an FS that doesn't support this throws an UnsupportedOperationException , not an IOException (see FileContext.java:1441 ). lines 1252 and 1313 : the contract of read() is not to read the whole file - that's the contract of readFully() . For this reason tests assuming that the whole file has been read should use readFully() instead of read() . And added an enhancement for object storage systems in the same file: line 1238 : an object storage system throws an IOException as a file does not exist before the file is closed (nor does it have a checksum at that moment). This object-storage issue is resolved by changing the order of fc.setVerifyChecksum(true, path) and out.write(data, 0, data.length) , while this does not impact the behaviour on hdfs or other file systems. Discovered and patched the following related bugs in S3A: Bugfix in S3AFileSystem.java : ports on s3 should be ignored, which corresponds with a value of -1 (instead of the default 0 in FileSystem). Another bugfix is in S3AFileStatus.java : getModificationTime() is overwritten for directories. It returns System.currentTimeMillis() because an ObjectStore does not keep track of modification-times of directories. Because some parts of the Hadoop ecosystem use modification time to ignore or delete "old" directories (e.g. the YarnHistorySever), returning 0 for directories is not the best option here. Added TestS3AMiniYarnCluster.java , which runs a simple WordCount -MapReduce job on a YarnMiniCluster using S3A as filesystem.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Patch 6:

          As requested, expanded patch 5 with tests by extending the following tests, overriding them with S3A specifics:

          • TestFileContext.java
          • FileContextCreateMkdirBaseTest.java
          • FileContextMainOperationsBaseTest.java
          • FCStatisticsBaseTest.java
          • FileContextURIBase.java
          • FileContextUtilBase.java

          In doing so, fixed following bugs in FileContextMainOperationsBaseTest.java:

          • line 1169: creating a symlink on an FS that doesn't support this throws an UnsupportedOperationException, not an IOException (see FileContext.java:1441).
          • lines 1252 and 1313: the contract of read() is not to read the whole file - that's the contract of readFully(). For this reason tests assuming that the whole file has been read should use readFully() instead of read().

          And added an enhancement for object storage systems in the same file:

          • line 1238: an object storage system throws an IOException as a file does not exist before the file is closed (nor does it have a checksum at that moment). This object-storage issue is resolved by changing the order of fc.setVerifyChecksum(true, path) and out.write(data, 0, data.length), while this does not impact the behaviour on hdfs or other file systems.

          Discovered and patched the following related bugs in S3A:

          • Bugfix in S3AFileSystem.java: ports on s3 should be ignored, which corresponds with a value of -1 (instead of the default 0 in FileSystem).
          • Another bugfix is in S3AFileStatus.java: getModificationTime() is overwritten for directories. It returns System.currentTimeMillis() because an ObjectStore does not keep track of modification-times of directories. Because some parts of the Hadoop ecosystem use modification time to ignore or delete "old" directories (e.g. the YarnHistorySever), returning 0 for directories is not the best option here.

          Added TestS3AMiniYarnCluster.java, which runs a simple WordCount-MapReduce job on a YarnMiniCluster using S3A as filesystem.

          Show
          PieterReuse Pieter Reuse added a comment - Patch 6: As requested, expanded patch 5 with tests by extending the following tests, overriding them with S3A specifics: TestFileContext.java FileContextCreateMkdirBaseTest.java FileContextMainOperationsBaseTest.java FCStatisticsBaseTest.java FileContextURIBase.java FileContextUtilBase.java In doing so, fixed following bugs in FileContextMainOperationsBaseTest.java : line 1169 : creating a symlink on an FS that doesn't support this throws an UnsupportedOperationException , not an IOException (see FileContext.java:1441 ). lines 1252 and 1313 : the contract of read() is not to read the whole file - that's the contract of readFully() . For this reason tests assuming that the whole file has been read should use readFully() instead of read() . And added an enhancement for object storage systems in the same file: line 1238 : an object storage system throws an IOException as a file does not exist before the file is closed (nor does it have a checksum at that moment). This object-storage issue is resolved by changing the order of fc.setVerifyChecksum(true, path) and out.write(data, 0, data.length) , while this does not impact the behaviour on hdfs or other file systems. Discovered and patched the following related bugs in S3A: Bugfix in S3AFileSystem.java : ports on s3 should be ignored, which corresponds with a value of -1 (instead of the default 0 in FileSystem). Another bugfix is in S3AFileStatus.java : getModificationTime() is overwritten for directories. It returns System.currentTimeMillis() because an ObjectStore does not keep track of modification-times of directories. Because some parts of the Hadoop ecosystem use modification time to ignore or delete "old" directories (e.g. the YarnHistorySever), returning 0 for directories is not the best option here. Added TestS3AMiniYarnCluster.java , which runs a simple WordCount -MapReduce job on a YarnMiniCluster using S3A as filesystem.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Whoops, sorry for the double post of this comment. You can ignore one of both.

          Show
          PieterReuse Pieter Reuse added a comment - Whoops, sorry for the double post of this comment. You can ignore one of both.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12743479/HADOOP-11262-6.patch
          Optional Tests javadoc javac unit findbugs checkstyle site
          git revision trunk / 71566e2
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7496/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743479/HADOOP-11262-6.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / 71566e2 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7496/console This message was automatically generated.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Uploaded patch version 7, a rebased version of patch 6 which did no longer apply.

          Other than that the only difference between versions 6 and 7 is that the first mentioned bug in S3AFileSystem is fixed in DelegateToFileSystem (see HADOOP-12304), so I removed the duplicate bugfix is in this patch.

          Show
          PieterReuse Pieter Reuse added a comment - Uploaded patch version 7, a rebased version of patch 6 which did no longer apply. Other than that the only difference between versions 6 and 7 is that the first mentioned bug in S3AFileSystem is fixed in DelegateToFileSystem (see HADOOP-12304 ), so I removed the duplicate bugfix is in this patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 11 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 3s Site still builds.
          -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 65, now 62).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 50s Tests failed in hadoop-common.
          +1 tools/hadoop tests 0m 14s Tests passed in hadoop-aws.
              71m 8s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestNetUtils



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751264/HADOOP-11262-7.patch
          Optional Tests javadoc javac unit findbugs checkstyle site
          git revision trunk / 2da5aaa
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/diffcheckstylehadoop-aws.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 11 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 3s Site still builds. -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 65, now 62). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 50s Tests failed in hadoop-common. +1 tools/hadoop tests 0m 14s Tests passed in hadoop-aws.     71m 8s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751264/HADOOP-11262-7.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / 2da5aaa checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/diffcheckstylehadoop-aws.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-common.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/console This message was automatically generated.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Test failures in hadoop-common unrelated. Patch still applies.

          Show
          PieterReuse Pieter Reuse added a comment - Test failures in hadoop-common unrelated. Patch still applies.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Pieter Reuse, thanks a lot for working on this. Could you update the code to obey Hadoop coding style (i.e., 2 space indent and putting @override below comments.). I think that we do not need @Override for @Before/@After/@Test methods.

               try {
                 // instruct FS to verify checksum through the FileContext:
          -      fc.setVerifyChecksum(true, path);
                 out.write(data, 0, data.length);
               } finally {
                 out.close();
          +      fc.setVerifyChecksum(true, path);
               }
          

          Could you help me to understand why you move setVerifyChecksum after close()? Wouldn't the checksum not being set in the write op?

          +    private String readStringFromFile(Path path){
          +        FSDataInputStream in = null;
          +        try {
          +            in = fs.open(path);
          +            long bytesLen = fs.getFileStatus(path).getLen();
          +            byte[] buffer = new byte[(int)bytesLen];
          +            IOUtils.readFully(in, buffer, 0, buffer.length);
          +            return new String(buffer);
          +        } catch (IOException e ){
          +            throw new RuntimeException("Failed to read from [" + path + "]", e);
          +        } finally {
          +            IOUtils.closeStream(in);
          +        }
          +    }
          

          Could you use try-with-resource in JDK7 for this? I see you have already used it in the other functions.

          Steve Loughran and Chris Nauroth, could you please provide some color on this JIRA? You have much more experiences on YARN.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Pieter Reuse , thanks a lot for working on this. Could you update the code to obey Hadoop coding style (i.e., 2 space indent and putting @override below comments.). I think that we do not need @Override for @Before/@After/@Test methods. try { // instruct FS to verify checksum through the FileContext: - fc.setVerifyChecksum( true , path); out.write(data, 0, data.length); } finally { out.close(); + fc.setVerifyChecksum( true , path); } Could you help me to understand why you move setVerifyChecksum after close() ? Wouldn't the checksum not being set in the write op? + private String readStringFromFile(Path path){ + FSDataInputStream in = null ; + try { + in = fs.open(path); + long bytesLen = fs.getFileStatus(path).getLen(); + byte [] buffer = new byte [( int )bytesLen]; + IOUtils.readFully(in, buffer, 0, buffer.length); + return new String (buffer); + } catch (IOException e ){ + throw new RuntimeException( "Failed to read from [" + path + "]" , e); + } finally { + IOUtils.closeStream(in); + } + } Could you use try-with-resource in JDK7 for this? I see you have already used it in the other functions. Steve Loughran and Chris Nauroth , could you please provide some color on this JIRA? You have much more experiences on YARN.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Lei (Eddy) Xu. Thanks for joining the code review.

          The reason this JIRA mentions enabling YARN to use S3A is that various points in the YARN codebase rely on the FileContext API instead of FileSystem. An example of this is storing container logs. It is possible to specify an HDFS URL in yarn.nodemanager.log-dirs, because HDFS provides a subclass of AbstractFileSystem. This is currently not possible with a S3A URL, because S3A does not provide a subclass of AbstractFileSystem.

          Show
          cnauroth Chris Nauroth added a comment - Hi Lei (Eddy) Xu . Thanks for joining the code review. The reason this JIRA mentions enabling YARN to use S3A is that various points in the YARN codebase rely on the FileContext API instead of FileSystem . An example of this is storing container logs. It is possible to specify an HDFS URL in yarn.nodemanager.log-dirs , because HDFS provides a subclass of AbstractFileSystem . This is currently not possible with a S3A URL, because S3A does not provide a subclass of AbstractFileSystem .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 3m 0s trunk passed
          +1 compile 4m 22s trunk passed with JDK v1.8.0_60
          +1 compile 4m 18s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 57s trunk passed
          +1 mvnsite 1m 17s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 17s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 46s the patch passed
          +1 compile 4m 22s the patch passed with JDK v1.8.0_60
          +1 javac 4m 22s the patch passed
          +1 compile 4m 14s the patch passed with JDK v1.7.0_79
          +1 javac 4m 14s the patch passed
          +1 checkstyle 0m 58s the patch passed
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 22s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 19s the patch passed with JDK v1.7.0_79
          +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60.
          +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.7.0_79.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          53m 47s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12751264/HADOOP-11262-7.patch
          JIRA Issue HADOOP-11262
          Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite
          uname Linux 67d9c13614bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / 5ba2b98
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8005/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8005/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 3m 0s trunk passed +1 compile 4m 22s trunk passed with JDK v1.8.0_60 +1 compile 4m 18s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 57s trunk passed +1 mvnsite 1m 17s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 46s the patch passed +1 compile 4m 22s the patch passed with JDK v1.8.0_60 +1 javac 4m 22s the patch passed +1 compile 4m 14s the patch passed with JDK v1.7.0_79 +1 javac 4m 14s the patch passed +1 checkstyle 0m 58s the patch passed +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 22s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 19s the patch passed with JDK v1.7.0_79 +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.8.0_60. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60. +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.7.0_79. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 53m 47s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12751264/HADOOP-11262-7.patch JIRA Issue HADOOP-11262 Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite uname Linux 67d9c13614bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / 5ba2b98 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8005/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8005/console This message was automatically generated.
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          Thanks for your time reviewing this. Chris Nauroth is correct. In hindsight, probably should have appended "as defaultFS" to the title of this jira. Feel free to adjust it.

          Lei (Eddy) Xu, Pieter Reuse is on holiday this week, we'll address your remarks next week.

          Show
          Thomas Demoor Thomas Demoor added a comment - Thanks for your time reviewing this. Chris Nauroth is correct. In hindsight, probably should have appended "as defaultFS" to the title of this jira. Feel free to adjust it. Lei (Eddy) Xu , Pieter Reuse is on holiday this week, we'll address your remarks next week.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks for the explanations, Chris Nauroth and Thomas Demoor.

          Looking forward to hear the updates.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks for the explanations, Chris Nauroth and Thomas Demoor . Looking forward to hear the updates.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Thank you, Lei (Eddy) Xu and Chris Nauroth for reviewing this and suggesting improvements. I've uploaded patch version 8, addressing Lei (Eddy) Xu's remarks about coding style, @Override in combination with @Before, @After or @Test and a try-with-recourses.

          Regarding the setVerifyChecksum-test, we (Thomas Demoor and I) noticed that the default FileSystem - and therefore S3A - simply ignores the "setVerifyChecksum"-flag, and that the test is therefore unnecessary for S3A. So overriding it with an empty method avoids having a falsely failing test.

          The original observation was that this flag is only used in the inputstream, and calling setVerifyChecksum() before out.close() is called makes S3A throw an IOException. Simply changing the order of these two lines fixed the issue we faced there, but patch 8 uses a different approach and does not change the testSetVerifyChecksum()-method in the super class any more.

          Show
          PieterReuse Pieter Reuse added a comment - Thank you, Lei (Eddy) Xu and Chris Nauroth for reviewing this and suggesting improvements. I've uploaded patch version 8, addressing Lei (Eddy) Xu 's remarks about coding style, @Override in combination with @Before, @After or @Test and a try-with-recourses. Regarding the setVerifyChecksum-test, we ( Thomas Demoor and I) noticed that the default FileSystem - and therefore S3A - simply ignores the "setVerifyChecksum"-flag, and that the test is therefore unnecessary for S3A. So overriding it with an empty method avoids having a falsely failing test. The original observation was that this flag is only used in the inputstream, and calling setVerifyChecksum() before out.close() is called makes S3A throw an IOException. Simply changing the order of these two lines fixed the issue we faced there, but patch 8 uses a different approach and does not change the testSetVerifyChecksum()-method in the super class any more.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 3m 21s trunk passed
          +1 compile 5m 19s trunk passed with JDK v1.8.0_60
          +1 compile 5m 5s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 15s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 24s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 34s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 52s the patch passed
          +1 compile 5m 21s the patch passed with JDK v1.8.0_60
          +1 javac 5m 21s the patch passed
          +1 compile 5m 3s the patch passed with JDK v1.7.0_79
          +1 javac 5m 3s the patch passed
          +1 checkstyle 1m 6s the patch passed
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 0s the patch passed
          +1 javadoc 1m 20s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 25s the patch passed with JDK v1.7.0_79
          -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.8.0_60.
          -1 unit 13m 7s hadoop-common in the patch failed with JDK v1.7.0_79.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          67m 23s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
          JDK v1.7.0_79 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.ipc.TestDecayRpcScheduler



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771363/HADOOP-11262-8.patch
          JIRA Issue HADOOP-11262
          Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite
          uname Linux 53f50af54fdb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / ef926b2
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 3m 21s trunk passed +1 compile 5m 19s trunk passed with JDK v1.8.0_60 +1 compile 5m 5s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 15s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 24s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 34s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 52s the patch passed +1 compile 5m 21s the patch passed with JDK v1.8.0_60 +1 javac 5m 21s the patch passed +1 compile 5m 3s the patch passed with JDK v1.7.0_79 +1 javac 5m 3s the patch passed +1 checkstyle 1m 6s the patch passed +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 0s the patch passed +1 javadoc 1m 20s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 25s the patch passed with JDK v1.7.0_79 -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_60. +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.8.0_60. -1 unit 13m 7s hadoop-common in the patch failed with JDK v1.7.0_79. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 67m 23s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag JDK v1.7.0_79 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.ipc.TestDecayRpcScheduler Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771363/HADOOP-11262-8.patch JIRA Issue HADOOP-11262 Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite uname Linux 53f50af54fdb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / ef926b2 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8054/console This message was automatically generated.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Test failures seem to be unrelated, I could not reproduce them locally and the code path of these tests is separate from this patch.
          Re-uploaded patch 8, forgot to remove the testSetVerifyChecksum()-changes yesterday.

          Show
          PieterReuse Pieter Reuse added a comment - Test failures seem to be unrelated, I could not reproduce them locally and the code path of these tests is separate from this patch. Re-uploaded patch 8, forgot to remove the testSetVerifyChecksum()-changes yesterday.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 3m 17s trunk passed
          +1 compile 4m 19s trunk passed with JDK v1.8.0_60
          +1 compile 4m 4s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 54s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 6s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 17s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 55s the patch passed
          +1 compile 4m 17s the patch passed with JDK v1.8.0_60
          +1 javac 4m 17s the patch passed
          +1 compile 4m 4s the patch passed with JDK v1.7.0_79
          +1 javac 4m 4s the patch passed
          +1 checkstyle 0m 55s the patch passed
          +1 mvnsite 1m 10s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 17s the patch passed with JDK v1.7.0_79
          -1 unit 6m 57s hadoop-common in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60.
          -1 unit 7m 26s hadoop-common in the patch failed with JDK v1.7.0_79.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          52m 55s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.7.0_79 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771566/HADOOP-11262-8.patch
          JIRA Issue HADOOP-11262
          Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite
          uname Linux 66f9c0beea5f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 0eb9c60
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 3m 17s trunk passed +1 compile 4m 19s trunk passed with JDK v1.8.0_60 +1 compile 4m 4s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 55s the patch passed +1 compile 4m 17s the patch passed with JDK v1.8.0_60 +1 javac 4m 17s the patch passed +1 compile 4m 4s the patch passed with JDK v1.7.0_79 +1 javac 4m 4s the patch passed +1 checkstyle 0m 55s the patch passed +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 25s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_79 -1 unit 6m 57s hadoop-common in the patch failed with JDK v1.8.0_60. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60. -1 unit 7m 26s hadoop-common in the patch failed with JDK v1.7.0_79. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 52m 55s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_79 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771566/HADOOP-11262-8.patch JIRA Issue HADOOP-11262 Optional Tests asflicense javac javadoc mvninstall unit xml findbugs checkstyle compile site mvnsite uname Linux 66f9c0beea5f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 0eb9c60 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8060/console This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Wanted to add an unofficial +1 - I've done quite a lot of testing on top of this patch and it looks good to me.

          Show
          mackrorysd Sean Mackrory added a comment - Wanted to add an unofficial +1 - I've done quite a lot of testing on top of this patch and it looks good to me.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. The last patch looks very good to me. Thanks for the work, Pieter Reuse and Thomas Demoor. And thanks for the reviews, Sean Mackrory.

          I will wait for Chris Nauroth's and Steve Loughran's opinions and will commit it tomorrow by the EOD if they are OK with this patch.

          Thanks guys.

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. The last patch looks very good to me. Thanks for the work, Pieter Reuse and Thomas Demoor . And thanks for the reviews, Sean Mackrory . I will wait for Chris Nauroth 's and Steve Loughran 's opinions and will commit it tomorrow by the EOD if they are OK with this patch. Thanks guys.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          LGTM, +1

          Lei -remember to bump up the year in the new copyright headers to 2016.

          Show
          stevel@apache.org Steve Loughran added a comment - LGTM, +1 Lei -remember to bump up the year in the new copyright headers to 2016.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v8 looks great overall. Thanks, everyone! I have just one question and one minor nitpick.

            @Override
            public long getModificationTime(){
              if(isDirectory()){
                return System.currentTimeMillis();
              } else {
                return super.getModificationTime();
              }
            }
          

          YARN resource localization includes a check of the modification time against the source file system. (See FSDownload#copy.) I believe the intent is to ensure that the state of the resources at application submission time remains the same at the time of launching the application's containers. Has anyone tested resource localization sourced from S3A? I expect an ever-changing modification time like this would trigger an error.

          I see you're using the strategy of overriding some irrelevant/unsupported test methods to be no-ops in the subclasses. Could you please put the @Ignore annotation on these?

          Show
          cnauroth Chris Nauroth added a comment - Patch v8 looks great overall. Thanks, everyone! I have just one question and one minor nitpick. @Override public long getModificationTime(){ if (isDirectory()){ return System .currentTimeMillis(); } else { return super .getModificationTime(); } } YARN resource localization includes a check of the modification time against the source file system. (See FSDownload#copy .) I believe the intent is to ensure that the state of the resources at application submission time remains the same at the time of launching the application's containers. Has anyone tested resource localization sourced from S3A? I expect an ever-changing modification time like this would trigger an error. I see you're using the strategy of overriding some irrelevant/unsupported test methods to be no-ops in the subclasses. Could you please put the @Ignore annotation on these?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          actually, cut the copyright 2015 line entirely. It goes into NOTICE.TXT and is updated in one place only. thanks

          Show
          stevel@apache.org Steve Loughran added a comment - actually, cut the copyright 2015 line entirely. It goes into NOTICE.TXT and is updated in one place only. thanks
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Steve Loughran Sure, will do. I pinged [~Pieter Reuse] and he said there will be a new patch to address Chirs' comments. I will commit after that.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Steve Loughran Sure, will do. I pinged [~Pieter Reuse] and he said there will be a new patch to address Chirs' comments. I will commit after that.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Thank you for the +1's, Sean Mackrory, Lei (Eddy) Xu and Steve Loughran.
          Chris Nauroth, thank you for looking at this patch.

          I have added @Ignore annotations where appropriate in version 9. I also removed the copyright lines featuring the year.

          Regarding the modification-times of directories in S3A: as directories are "fakes" in s3a, there is no feasible way to get accurate directory timestamps without extensive locking (and coping with slow listings), which counters the rationale of object stores. Therefore, we chose a "dummy" implementation that doesn't break (too many) things.

          Setting a fixed time (e.g. epoch) breaks the history server as it looks at the modification time of the directory-object before moving it, and decides the files don't need to be copied if they are "too old". Setting the modificationtime of directories in S3A to System.currentTimeMillis() ensures the historyserver never labels them as being "too old".

          Good that you have taken a deeper look into whether always labelling directories as "too young" can give rise to problems in YARN. Looking deeper into the classes LocalResource and LocalResourceType learns that the YARN resource localization is always executed against regular files or .jar-archives (these are the only possible values of LocalResourceType), for which S3A returns the correct timestamps.

          However, looking at the AggregatedLogDeletionService of YARN learns that this service will omit removing the appropriate logfiles because the directory will be labelled "too young". I did not find any other places in YARN where this patch can cause problems. I indicated this behaviour in the index.md file in the patch. As this is no breaking situation, I still propose to go forward with this patch.

          Show
          PieterReuse Pieter Reuse added a comment - Thank you for the +1's, Sean Mackrory , Lei (Eddy) Xu and Steve Loughran . Chris Nauroth , thank you for looking at this patch. I have added @Ignore annotations where appropriate in version 9. I also removed the copyright lines featuring the year. Regarding the modification-times of directories in S3A: as directories are "fakes" in s3a, there is no feasible way to get accurate directory timestamps without extensive locking (and coping with slow listings), which counters the rationale of object stores. Therefore, we chose a "dummy" implementation that doesn't break (too many) things. Setting a fixed time (e.g. epoch) breaks the history server as it looks at the modification time of the directory-object before moving it, and decides the files don't need to be copied if they are "too old". Setting the modificationtime of directories in S3A to System.currentTimeMillis() ensures the historyserver never labels them as being "too old". Good that you have taken a deeper look into whether always labelling directories as "too young" can give rise to problems in YARN. Looking deeper into the classes LocalResource and LocalResourceType learns that the YARN resource localization is always executed against regular files or .jar-archives (these are the only possible values of LocalResourceType), for which S3A returns the correct timestamps. However, looking at the AggregatedLogDeletionService of YARN learns that this service will omit removing the appropriate logfiles because the directory will be labelled "too young". I did not find any other places in YARN where this patch can cause problems. I indicated this behaviour in the index.md file in the patch. As this is no breaking situation, I still propose to go forward with this patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          Pieter Reuse, thanks for diving deep into the practical usage of mtime in that last response. I agree with the trade-off. Let's proceed with this.

          I see just one more test that needs a @Ignore annotation: TestS3AFileContextMainOperations#testSetVerifyChecksum. I'll be +1 after that.

          Show
          cnauroth Chris Nauroth added a comment - Pieter Reuse , thanks for diving deep into the practical usage of mtime in that last response. I agree with the trade-off. Let's proceed with this. I see just one more test that needs a @Ignore annotation: TestS3AFileContextMainOperations#testSetVerifyChecksum . I'll be +1 after that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 7m 53s trunk passed
          +1 compile 8m 36s trunk passed with JDK v1.8.0_66
          +1 compile 9m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 25s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 22s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 52s the patch passed
          +1 compile 8m 31s the patch passed with JDK v1.8.0_66
          +1 javac 8m 31s the patch passed
          +1 compile 9m 18s the patch passed with JDK v1.7.0_91
          +1 javac 9m 18s the patch passed
          +1 checkstyle 1m 2s the patch passed
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 44s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 21s the patch passed with JDK v1.7.0_91
          -1 unit 7m 46s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 18s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          80m 11s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781472/HADOOP-11262-9.patch
          JIRA Issue HADOOP-11262
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9fa7ff1773be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0e76f1f
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 7m 53s trunk passed +1 compile 8m 36s trunk passed with JDK v1.8.0_66 +1 compile 9m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 25s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 22s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 52s the patch passed +1 compile 8m 31s the patch passed with JDK v1.8.0_66 +1 javac 8m 31s the patch passed +1 compile 9m 18s the patch passed with JDK v1.7.0_91 +1 javac 9m 18s the patch passed +1 checkstyle 1m 2s the patch passed +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 44s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 21s the patch passed with JDK v1.7.0_91 -1 unit 7m 46s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66. +1 unit 8m 18s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 80m 11s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781472/HADOOP-11262-9.patch JIRA Issue HADOOP-11262 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9fa7ff1773be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0e76f1f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8378/console This message was automatically generated.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Thank you for pointing out the missing @Ignore, Chris Nauroth. Added this to patch version 10.

          Test failures of Hadoop QA on patch version 9 are unrelated.

          Show
          PieterReuse Pieter Reuse added a comment - Thank you for pointing out the missing @Ignore, Chris Nauroth . Added this to patch version 10. Test failures of Hadoop QA on patch version 9 are unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 8m 0s trunk passed
          +1 compile 8m 55s trunk passed with JDK v1.8.0_66
          +1 compile 9m 28s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 27s trunk passed
          +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 24s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 52s the patch passed
          +1 compile 8m 49s the patch passed with JDK v1.8.0_66
          +1 javac 8m 49s the patch passed
          +1 compile 9m 29s the patch passed with JDK v1.7.0_91
          +1 javac 9m 29s the patch passed
          +1 checkstyle 1m 7s the patch passed
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 48s the patch passed
          +1 javadoc 1m 13s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 24s the patch passed with JDK v1.7.0_91
          -1 unit 7m 56s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_66.
          -1 unit 0m 38s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          73m 59s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781527/HADOOP-11262-10.patch
          JIRA Issue HADOOP-11262
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 22d298f08859 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 76e7270
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 8m 0s trunk passed +1 compile 8m 55s trunk passed with JDK v1.8.0_66 +1 compile 9m 28s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 27s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 52s the patch passed +1 compile 8m 49s the patch passed with JDK v1.8.0_66 +1 javac 8m 49s the patch passed +1 compile 9m 29s the patch passed with JDK v1.7.0_91 +1 javac 9m 29s the patch passed +1 checkstyle 1m 7s the patch passed +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 48s the patch passed +1 javadoc 1m 13s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 24s the patch passed with JDK v1.7.0_91 -1 unit 7m 56s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_66. -1 unit 0m 38s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 73m 59s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781527/HADOOP-11262-10.patch JIRA Issue HADOOP-11262 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 22d298f08859 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 76e7270 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8381/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v10. I'll hold off committing until end of day in case anyone else wants to comment again.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v10. I'll hold off committing until end of day in case anyone else wants to comment again.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. Thanks a lot for the great work here, Pieter Reuse and Thomas Demoor. Also thanks for the reviews from Chris Nauroth, Steve Loughran and Sean Mackrory.

          I committed this patch to trunk, branch-2 and branch-2.8.

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. Thanks a lot for the great work here, Pieter Reuse and Thomas Demoor . Also thanks for the reviews from Chris Nauroth , Steve Loughran and Sean Mackrory . I committed this patch to trunk, branch-2 and branch-2.8.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9097/)
          HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) (lei: rev 126705f67eaa6d866f7572fbddf133c5d7552353)

          • hadoop-tools/hadoop-aws/src/test/resources/core-site.xml
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextURI.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3AMiniYarnCluster.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContext.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3A.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextCreateMkdir.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextMainOperations.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextStatistics.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3A.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9097/ ) HADOOP-11262 . Enable YARN to use S3A. (Pieter Reuse via lei) (lei: rev 126705f67eaa6d866f7572fbddf133c5d7552353) hadoop-tools/hadoop-aws/src/test/resources/core-site.xml hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextURI.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3AMiniYarnCluster.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContext.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3A.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextCreateMkdir.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextMainOperations.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextStatistics.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3A.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Lei (Eddy) Xu, thank you for taking care of the commit.

          Show
          cnauroth Chris Nauroth added a comment - Lei (Eddy) Xu , thank you for taking care of the commit.
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          Thank you for supporting this feature and for your reviews, advice and help Lei (Eddy) Xu, Chris Nauroth, Steve Loughran, Sean Mackrory.

          Ping Jakob Homan: you showed initial interest in this, simply notifying you that it's been merged in.

          Show
          Thomas Demoor Thomas Demoor added a comment - Thank you for supporting this feature and for your reviews, advice and help Lei (Eddy) Xu , Chris Nauroth , Steve Loughran , Sean Mackrory . Ping Jakob Homan : you showed initial interest in this, simply notifying you that it's been merged in.

            People

            • Assignee:
              PieterReuse Pieter Reuse
              Reporter:
              thodemoor Thomas Demoor
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development