Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12851

S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-12846 introduced the ability for FileSystem based integration points of credential providers to eliminate the threat of a recursive infinite loop due to a provider in the same filesystem being configured.

      It was WASB has already uptaken its use in HADOOP-12846 and this patch will add it to the S3A integration point as well.

      1. HADOOP-12851-001.patch
        10 kB
        Larry McCay
      2. HADOOP-12851-002.patch
        10 kB
        Larry McCay
      3. HADOOP-12851-003.patch
        11 kB
        Larry McCay
      4. HADOOP-12851-004.patch
        11 kB
        Larry McCay

        Issue Links

          Activity

          Hide
          lmccay Larry McCay added a comment -

          Thanks, Chris Nauroth!

          Show
          lmccay Larry McCay added a comment - Thanks, Chris Nauroth !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9400/)
          HADOOP-12851. S3AFileSystem Uptake of (cnauroth: rev d251e55415f1fab085159b9eb3b43214d100b6a8)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9400/ ) HADOOP-12851 . S3AFileSystem Uptake of (cnauroth: rev d251e55415f1fab085159b9eb3b43214d100b6a8) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v004. I have committed this to trunk, branch-2 and branch-2.8. Larry McCay, thank you for the patch.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v004. I have committed this to trunk, branch-2 and branch-2.8. Larry McCay , thank you for the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 13s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_72
          +1 compile 0m 15s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 9s the patch passed with JDK v1.8.0_72
          +1 javac 0m 9s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.7.0_95
          +1 javac 0m 11s the patch passed
          +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 0 new + 38 unchanged - 2 fixed = 38 total (was 40)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 40s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 9s hadoop-aws in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          14m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790629/HADOOP-12851-004.patch
          JIRA Issue HADOOP-12851
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6948804ac908 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d8f390d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8753/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8753/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 13s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_72 +1 compile 0m 15s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 15s the patch passed +1 compile 0m 9s the patch passed with JDK v1.8.0_72 +1 javac 0m 9s the patch passed +1 compile 0m 11s the patch passed with JDK v1.7.0_95 +1 javac 0m 11s the patch passed +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 0 new + 38 unchanged - 2 fixed = 38 total (was 40) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 9s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 14m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790629/HADOOP-12851-004.patch JIRA Issue HADOOP-12851 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6948804ac908 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8f390d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8753/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8753/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          v004 of the patch provisions the keys to the expected credential provider in TestS3AConfiguration.

          Show
          lmccay Larry McCay added a comment - v004 of the patch provisions the keys to the expected credential provider in TestS3AConfiguration.
          Hide
          lmccay Larry McCay added a comment -

          Ahhh - that makes sense - yeah we can use the modified config for the provisioning - it just grabs the first provider in the path and adds the credentials to it. So, it appears to think that it is adding it to the phoney jceks://s3a/foobar provider which is then removed from the config from the later getPassword integration.

          The fact that it seems to be adding the creds to the jceks://s3a/foobar provider, it seems that it is being removed from the getPassword in order to access the s3a filesystem too.

          For clarity, I will add another version though which adds it to the one with the providerPath that doesn't include s3a based provider.

          Show
          lmccay Larry McCay added a comment - Ahhh - that makes sense - yeah we can use the modified config for the provisioning - it just grabs the first provider in the path and adds the credentials to it. So, it appears to think that it is adding it to the phoney jceks://s3a/foobar provider which is then removed from the config from the later getPassword integration. The fact that it seems to be adding the creds to the jceks://s3a/foobar provider, it seems that it is being removed from the getPassword in order to access the s3a filesystem too. For clarity, I will add another version though which adds it to the one with the providerPath that doesn't include s3a based provider.
          Hide
          cnauroth Chris Nauroth added a comment -

          Larry McCay, in patch v03, in TestS3AConfiguration, is this line...

              provisionAccessKeys(conf);
          

          ...supposed to use the modified configuration instead?

              provisionAccessKeys(c);
          

          Currently, I see the failure shown below, but the test passes if I make that change.

          Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.594 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          testExcludingS3ACredentialProvider(org.apache.hadoop.fs.s3a.TestS3AConfiguration)  Time elapsed: 1.461 sec  <<< ERROR!
          java.lang.IllegalArgumentException: The bucketName parameter must be specified.
          	at com.amazonaws.services.s3.AmazonS3Client.rejectNull(AmazonS3Client.java:3032)
          	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1102)
          	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:177)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2790)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2827)
          	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2809)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381)
          	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294)
          	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.initFileSystem(JavaKeyStoreProvider.java:89)
          	at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:82)
          	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49)
          	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41)
          	at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
          	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
          	at org.apache.hadoop.fs.s3a.TestS3AConfiguration.provisionAccessKeys(TestS3AConfiguration.java:221)
          	at org.apache.hadoop.fs.s3a.TestS3AConfiguration.testExcludingS3ACredentialProvider(TestS3AConfiguration.java:342)
          
          Show
          cnauroth Chris Nauroth added a comment - Larry McCay , in patch v03, in TestS3AConfiguration , is this line... provisionAccessKeys(conf); ...supposed to use the modified configuration instead? provisionAccessKeys(c); Currently, I see the failure shown below, but the test passes if I make that change. Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.594 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration testExcludingS3ACredentialProvider(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 1.461 sec <<< ERROR! java.lang.IllegalArgumentException: The bucketName parameter must be specified. at com.amazonaws.services.s3.AmazonS3Client.rejectNull(AmazonS3Client.java:3032) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1102) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:177) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2790) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2827) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2809) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.initFileSystem(JavaKeyStoreProvider.java:89) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:82) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41) at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58) at org.apache.hadoop.fs.s3a.TestS3AConfiguration.provisionAccessKeys(TestS3AConfiguration.java:221) at org.apache.hadoop.fs.s3a.TestS3AConfiguration.testExcludingS3ACredentialProvider(TestS3AConfiguration.java:342)
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 16m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 21s trunk passed
          +1 compile 0m 11s trunk passed with JDK v1.8.0_72
          +1 compile 0m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 9s the patch passed with JDK v1.8.0_72
          +1 javac 0m 9s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.7.0_95
          +1 javac 0m 12s the patch passed
          +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 0 new + 38 unchanged - 2 fixed = 38 total (was 40)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 9s hadoop-aws in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          30m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790602/HADOOP-12851-003.patch
          JIRA Issue HADOOP-12851
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9246ed879173 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9dafaaa
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8751/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8751/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 16m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 21s trunk passed +1 compile 0m 11s trunk passed with JDK v1.8.0_72 +1 compile 0m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 16s the patch passed +1 compile 0m 9s the patch passed with JDK v1.8.0_72 +1 javac 0m 9s the patch passed +1 compile 0m 12s the patch passed with JDK v1.7.0_95 +1 javac 0m 12s the patch passed +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 0 new + 38 unchanged - 2 fixed = 38 total (was 40) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 9s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 30m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790602/HADOOP-12851-003.patch JIRA Issue HADOOP-12851 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9246ed879173 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9dafaaa Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8751/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8751/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 39s trunk passed
          +1 compile 0m 12s trunk passed with JDK v1.8.0_72
          +1 compile 0m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 9s the patch passed with JDK v1.8.0_72
          +1 javac 0m 9s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.7.0_95
          +1 javac 0m 11s the patch passed
          -1 checkstyle 0m 11s hadoop-tools/hadoop-aws: patch generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40)
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          25m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790589/HADOOP-12851-002.patch
          JIRA Issue HADOOP-12851
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 322ecbb0b2e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0fa54d4
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 39s trunk passed +1 compile 0m 12s trunk passed with JDK v1.8.0_72 +1 compile 0m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 13s the patch passed +1 compile 0m 9s the patch passed with JDK v1.8.0_72 +1 javac 0m 9s the patch passed +1 compile 0m 11s the patch passed with JDK v1.7.0_95 +1 javac 0m 11s the patch passed -1 checkstyle 0m 11s hadoop-tools/hadoop-aws: patch generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40) +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 25m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790589/HADOOP-12851-002.patch JIRA Issue HADOOP-12851 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 322ecbb0b2e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fa54d4 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8749/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          v003 addresses the compilation errors found by Chris Nauroth and fixes the indentation problem indicated by precommit on an earlier patch.

          Show
          lmccay Larry McCay added a comment - v003 addresses the compilation errors found by Chris Nauroth and fixes the indentation problem indicated by precommit on an earlier patch.
          Hide
          lmccay Larry McCay added a comment -

          v002 is same as it ever was - I did make sure that it builds and tests pass again though. Very weird that you are seeing that.

          Show
          lmccay Larry McCay added a comment - v002 is same as it ever was - I did make sure that it builds and tests pass again though. Very weird that you are seeing that.
          Hide
          cnauroth Chris Nauroth added a comment -

          Strange... we clearly had a good Jenkins run, but when I apply the patch, I get compilation errors. Larry McCay, could you please upload a new patch? Maybe something awkward happened with the JIRA attachments.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-aws: Compilation failure: Compilation failure:
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java:[334,15] cannot find symbol
          [ERROR] symbol:   variable S3AFileSystem
          [ERROR] location: class org.apache.hadoop.fs.s3a.TestS3AConfiguration
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java:[337,5] cannot find symbol
          [ERROR] symbol:   method assertFalse(java.lang.String,boolean)
          [ERROR] location: class org.apache.hadoop.fs.s3a.TestS3AConfiguration
          
          Show
          cnauroth Chris Nauroth added a comment - Strange... we clearly had a good Jenkins run, but when I apply the patch, I get compilation errors. Larry McCay , could you please upload a new patch? Maybe something awkward happened with the JIRA attachments. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile ( default -testCompile) on project hadoop-aws: Compilation failure: Compilation failure: [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java:[334,15] cannot find symbol [ERROR] symbol: variable S3AFileSystem [ERROR] location: class org.apache.hadoop.fs.s3a.TestS3AConfiguration [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java:[337,5] cannot find symbol [ERROR] symbol: method assertFalse(java.lang. String , boolean ) [ERROR] location: class org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Hide
          lmccay Larry McCay added a comment -

          Chris Nauroth - when you have a couple free cycles can you give this a quick review - it is a follow up on HADOOP-12846.

          Show
          lmccay Larry McCay added a comment - Chris Nauroth - when you have a couple free cycles can you give this a quick review - it is a follow up on HADOOP-12846 .
          Hide
          lmccay Larry McCay added a comment -

          Checkstyle issue reported above is existing code and my patches actually fixes numerous lines with trailing whitespace violations. I'd rather this not block the patch. If after some review that changes are required then I can address that one too.

          Show
          lmccay Larry McCay added a comment - Checkstyle issue reported above is existing code and my patches actually fixes numerous lines with trailing whitespace violations. I'd rather this not block the patch. If after some review that changes are required then I can address that one too.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 41s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_72
          +1 compile 0m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 18s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 23s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.8.0_72
          +1 javac 0m 18s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_95
          +1 javac 0m 16s the patch passed
          -1 checkstyle 0m 17s hadoop-tools/hadoop-aws: patch generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40)
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 0m 20s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95
          +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          16m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790513/HADOOP-12851-001.patch
          JIRA Issue HADOOP-12851
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 27bfb79b1b9e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1cb2f93
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_72 +1 compile 0m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 23s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 22s the patch passed +1 compile 0m 18s the patch passed with JDK v1.8.0_72 +1 javac 0m 18s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_95 +1 javac 0m 16s the patch passed -1 checkstyle 0m 17s hadoop-tools/hadoop-aws: patch generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95 +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 16m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790513/HADOOP-12851-001.patch JIRA Issue HADOOP-12851 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27bfb79b1b9e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1cb2f93 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8745/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          lmccay Larry McCay added a comment -

          Initial patch to uptake excludeIncompatibleCredentialProviders and avoid recursive dependencies.

          Show
          lmccay Larry McCay added a comment - Initial patch to uptake excludeIncompatibleCredentialProviders and avoid recursive dependencies.

            People

            • Assignee:
              lmccay Larry McCay
              Reporter:
              lmccay Larry McCay
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development