Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      S3A now supports configuration of multiple credential provider classes for authenticating to S3. These are loaded and queried in sequence for a valid set of credentials. For more details, refer to the description of the fs.s3a.aws.credentials.provider configuration property or the S3A documentation page.
      Show
      S3A now supports configuration of multiple credential provider classes for authenticating to S3. These are loaded and queried in sequence for a valid set of credentials. For more details, refer to the description of the fs.s3a.aws.credentials.provider configuration property or the S3A documentation page.

      Description

      We've now got some fairly complex auth mechanisms going on: -hadoop config, KMS, env vars, "none". IF something isn't working, it's going to be a lot harder to debug.

      Review and tune the S3A provider point

      • add logging of what's going on in s3 auth to help debug problems
      • make a whole chain of logins expressible
      • allow the anonymous credentials to be included in the list
      • review and updated documents.

      I propose carefully adding some debug messages to identify which auth provider is doing the auth, so we can see if the env vars were kicking in, sysprops, etc.

      What we mustn't do is leak any secrets: this should be identifying whether properties and env vars are set, not what their values are. I don't believe that this will generate a security risk.

      1. HADOOP-13252-006.patch
        52 kB
        Steve Loughran
      2. HADOOP-13252-007.patch
        55 kB
        Steve Loughran
      3. HADOOP-13252-branch-2-001.patch
        34 kB
        Steve Loughran
      4. HADOOP-13252-branch-2-003.patch
        46 kB
        Steve Loughran
      5. HADOOP-13252-branch-2-004.patch
        46 kB
        Steve Loughran
      6. HADOOP-13252-branch-2-005.patch
        49 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          lmccay Larry McCay added a comment -

          Steve Loughran - by auth mechanisms, it seems that you mean mechanisms for looking up credentials for auth. Correct? I'd just like to point out that indicating "none" is leaking a secret - whether this be done explicitly or implicitly.

          We may want to leave this to indicating whether hadoop config of the credentials or a credential provider is being used. Not the level of detail that you are looking for but it would at least point someone with proper permissions to read the config to the right place.

          Show
          lmccay Larry McCay added a comment - Steve Loughran - by auth mechanisms, it seems that you mean mechanisms for looking up credentials for auth. Correct? I'd just like to point out that indicating "none" is leaking a secret - whether this be done explicitly or implicitly. We may want to leave this to indicating whether hadoop config of the credentials or a credential provider is being used. Not the level of detail that you are looking for but it would at least point someone with proper permissions to read the config to the right place.
          Hide
          cnauroth Chris Nauroth added a comment -

          One thing that's helpful, and works now without code changes, is to add this to log4j.properties:

          log4j.logger.com.amazonaws.auth=DEBUG
          

          Then the AWS SDK will print which AWS credential provider was selected from the chain, without leaking any secrets.

          > hadoop fs -ls s3a://cnauroth-test-aws-s3a/
          16/06/09 13:14:13 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider
          

          However, if this is more about Hadoop's credential provider API, then I don't think it has the logging you're looking for yet.

          Show
          cnauroth Chris Nauroth added a comment - One thing that's helpful, and works now without code changes, is to add this to log4j.properties: log4j.logger.com.amazonaws.auth=DEBUG Then the AWS SDK will print which AWS credential provider was selected from the chain, without leaking any secrets. > hadoop fs -ls s3a: //cnauroth-test-aws-s3a/ 16/06/09 13:14:13 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider However, if this is more about Hadoop's credential provider API, then I don't think it has the logging you're looking for yet.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I looked at that, but I also noticed it's logging of auth failures was only exception.getMessage(), not the full stack.

          Show
          stevel@apache.org Steve Loughran added a comment - I looked at that, but I also noticed it's logging of auth failures was only exception.getMessage(), not the full stack.
          Hide
          cnauroth Chris Nauroth added a comment -

          There really isn't very much code to AWSCredentialsProviderChain:

          https://github.com/aws/aws-sdk-java/blob/1.10.6/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java

          It's pretty much just a list of the underlying providers and a loop to call getCredentials() on each one in sequence. If we'd prefer different logging, I don't think it would be too cumbersome to make our own chain implementation and use that instead.

          Show
          cnauroth Chris Nauroth added a comment - There really isn't very much code to AWSCredentialsProviderChain : https://github.com/aws/aws-sdk-java/blob/1.10.6/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java It's pretty much just a list of the underlying providers and a loop to call getCredentials() on each one in sequence. If we'd prefer different logging, I don't think it would be too cumbersome to make our own chain implementation and use that instead.
          Hide
          cnauroth Chris Nauroth added a comment -

          ...and perhaps simultaneously file a pull request to the AWS SDK so that we can drop our custom version in a later release.

          Show
          cnauroth Chris Nauroth added a comment - ...and perhaps simultaneously file a pull request to the AWS SDK so that we can drop our custom version in a later release.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001

          1. review credential use; move loader code into S3AUtils for ease of testing
          2. TemporaryCredentials to use Hadoop Credential Provider API
          3. Add a SimpleAWSCredentialsProvider which supports basic credentials and Can be listed in fs.s3a.aws.credentials.provider
          4. fs.s3a.aws.credentials.provider takes a list of classes
          5. Add AWSCredentialProviderList. This AWSCredentialsProviderChain with
          • Allows extra providers to be added dynamically.
          • If any provider in the chain throws an exception other than an AmazonClientException, that is rethrown, rather than swallowed
          • Has some more diagnostics.
          • On failure, the last AmazonClientException raised is rethrown.
          • Special handling of AnonymousAWSCredentials. (i.e. supported in the chain)
          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001 review credential use; move loader code into S3AUtils for ease of testing TemporaryCredentials to use Hadoop Credential Provider API Add a SimpleAWSCredentialsProvider which supports basic credentials and Can be listed in fs.s3a.aws.credentials.provider fs.s3a.aws.credentials.provider takes a list of classes Add AWSCredentialProviderList. This AWSCredentialsProviderChain with Allows extra providers to be added dynamically. If any provider in the chain throws an exception other than an AmazonClientException, that is rethrown, rather than swallowed Has some more diagnostics. On failure, the last AmazonClientException raised is rethrown. Special handling of AnonymousAWSCredentials. (i.e. supported in the chain)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 10m 30s Docker failed to build yetus/hadoop:fb13ab0.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812918/HADOOP-13252-branch-2-001.patch
          JIRA Issue HADOOP-13252
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9861/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 10m 30s Docker failed to build yetus/hadoop:fb13ab0. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812918/HADOOP-13252-branch-2-001.patch JIRA Issue HADOOP-13252 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9861/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 003; rebase to trunk and rebuild to trigger yetus off

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 003; rebase to trunk and rebuild to trigger yetus off
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 36s Maven dependency ordering for branch
          +1 mvninstall 7m 12s branch-2 passed
          +1 compile 6m 43s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 31s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 20s branch-2 passed
          +1 mvnsite 1m 20s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 2m 14s branch-2 passed
          +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 5m 48s the patch passed with JDK v1.8.0_91
          +1 javac 5m 48s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_101
          +1 javac 6m 32s the patch passed
          -0 checkstyle 1m 23s root: The patch generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 1s The patch 2 line(s) with tabs.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 42s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 28s the patch passed with JDK v1.7.0_101
          +1 unit 9m 35s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          96m 27s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814107/HADOOP-13252-branch-2-003.patch
          JIRA Issue HADOOP-13252
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3e86d2629124 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 1e34763
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 36s Maven dependency ordering for branch +1 mvninstall 7m 12s branch-2 passed +1 compile 6m 43s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 31s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 20s branch-2 passed +1 mvnsite 1m 20s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 2m 14s branch-2 passed +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 5m 48s the patch passed with JDK v1.8.0_91 +1 javac 5m 48s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_101 +1 javac 6m 32s the patch passed -0 checkstyle 1m 23s root: The patch generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 1s The patch 2 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 42s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_101 +1 unit 9m 35s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 96m 27s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814107/HADOOP-13252-branch-2-003.patch JIRA Issue HADOOP-13252 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3e86d2629124 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1e34763 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9883/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 004; rebased to branch 2; the STS test changes had broken the merge

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 004; rebased to branch 2; the STS test changes had broken the merge
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 45s Maven dependency ordering for branch
          +1 mvninstall 6m 23s branch-2 passed
          +1 compile 5m 27s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 25s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 21s branch-2 passed
          +1 mvnsite 1m 17s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 2m 14s branch-2 passed
          +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 6m 9s the patch passed with JDK v1.8.0_91
          +1 javac 6m 9s the patch passed
          +1 compile 6m 37s the patch passed with JDK v1.7.0_101
          +1 javac 6m 37s the patch passed
          -0 checkstyle 1m 24s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 2 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 46s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101
          +1 unit 7m 28s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          91m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819967/HADOOP-13252-branch-2-004.patch
          JIRA Issue HADOOP-13252
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 29b44e6512cf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / b757eff
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 6m 23s branch-2 passed +1 compile 5m 27s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 25s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 21s branch-2 passed +1 mvnsite 1m 17s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 2m 14s branch-2 passed +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 6m 9s the patch passed with JDK v1.8.0_91 +1 javac 6m 9s the patch passed +1 compile 6m 37s the patch passed with JDK v1.7.0_101 +1 javac 6m 37s the patch passed -0 checkstyle 1m 24s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 2 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 46s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101 +1 unit 7m 28s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 91m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819967/HADOOP-13252-branch-2-004.patch JIRA Issue HADOOP-13252 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 29b44e6512cf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / b757eff Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10077/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          The patch looks good. I have just a few minor comments.

          In core-default.xml, please mention that the list of credentials provider classes is comma-separated.

          Please add visibility/stability annotations to AWSCredentialProviderList.

          which integrate with the AWS SDK by implementing the `om.amazonaws.auth.AWSCredentialsProvider`.
          

          Typo in class name.

          1. Alowing anonymous access to an S3 bucket compromises
          

          Typo: "Allowing"

          from placing its declaration on the commant line.
          

          Typo: "command"

          Show
          cnauroth Chris Nauroth added a comment - The patch looks good. I have just a few minor comments. In core-default.xml, please mention that the list of credentials provider classes is comma-separated. Please add visibility/stability annotations to AWSCredentialProviderList . which integrate with the AWS SDK by implementing the `om.amazonaws.auth.AWSCredentialsProvider`. Typo in class name. 1. Alowing anonymous access to an S3 bucket compromises Typo: "Allowing" from placing its declaration on the commant line. Typo: "command"
          Hide
          cnauroth Chris Nauroth added a comment -

          In core-default.xml, please mention that the list of credentials provider classes is comma-separated.

          This comment also applies to the copy of the content in index.md.

          Show
          cnauroth Chris Nauroth added a comment - In core-default.xml, please mention that the list of credentials provider classes is comma-separated. This comment also applies to the copy of the content in index.md.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 005. Rebased to branch-2, address chris's comments, do another proofread of the markdown file.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 005. Rebased to branch-2, address chris's comments, do another proofread of the markdown file.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 4s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 18m 29s Maven dependency ordering for branch
          +1 mvninstall 6m 47s branch-2 passed
          +1 compile 5m 28s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 24s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 23s branch-2 passed
          +1 mvnsite 1m 21s branch-2 passed
          +1 mvneclipse 1m 27s branch-2 passed
          +1 findbugs 2m 23s branch-2 passed
          +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 5m 24s the patch passed with JDK v1.8.0_101
          +1 javac 5m 24s the patch passed
          +1 compile 6m 28s the patch passed with JDK v1.7.0_111
          +1 javac 6m 28s the patch passed
          -0 checkstyle 1m 25s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 52 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 49s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_111
          +1 unit 8m 8s hadoop-common in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          123m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824152/HADOOP-13252-branch-2-005.patch
          JIRA Issue HADOOP-13252
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 567853706ec2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 3ee1fb7
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 4s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 18m 29s Maven dependency ordering for branch +1 mvninstall 6m 47s branch-2 passed +1 compile 5m 28s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 24s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 23s branch-2 passed +1 mvnsite 1m 21s branch-2 passed +1 mvneclipse 1m 27s branch-2 passed +1 findbugs 2m 23s branch-2 passed +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 5m 24s the patch passed with JDK v1.8.0_101 +1 javac 5m 24s the patch passed +1 compile 6m 28s the patch passed with JDK v1.7.0_111 +1 javac 6m 28s the patch passed -0 checkstyle 1m 25s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 52 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 49s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_111 +1 unit 8m 8s hadoop-common in the patch passed with JDK v1.7.0_111. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 123m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824152/HADOOP-13252-branch-2-005.patch JIRA Issue HADOOP-13252 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 567853706ec2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 3ee1fb7 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10281/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch 005 for branch-2 looks good after addressing the Checkstyle nitpicks.

          However, I am seeing test failures when I apply this patch to trunk. (Full stack trace below.) It seems like getting the classes out of Configuration is behaving differently on trunk vs. branch-2.

                awsClasses = conf.getClasses(AWS_CREDENTIALS_PROVIDER,
                    AWSCredentialsProvider.class);
          

          The difference in behavior might be due to HADOOP-7851, which I notice is in trunk but not branch-2.

          Steve, would you please take a look? AWSCredentialsProvider can't be used as the default, because it can't be instantiated.

          testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate)  Time elapsed: 1.362 sec  <<< ERROR!
          java.io.IOException: com.amazonaws.auth.AWSCredentialsProvider constructor exception.  A class specified in fs.s3a.aws.credentials.provider must provide an accessible constructor accepting URI and Configuration, or an accessible default constructor.
          	at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:319)
          	at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842)
          	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388)
          	at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
          	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Caused by: java.lang.NoSuchMethodException: com.amazonaws.auth.AWSCredentialsProvider.<init>()
          	at java.lang.Class.getConstructor0(Class.java:3082)
          	at java.lang.Class.getDeclaredConstructor(Class.java:2178)
          	at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:315)
          	at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842)
          	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388)
          	at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
          	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          cnauroth Chris Nauroth added a comment - Patch 005 for branch-2 looks good after addressing the Checkstyle nitpicks. However, I am seeing test failures when I apply this patch to trunk. (Full stack trace below.) It seems like getting the classes out of Configuration is behaving differently on trunk vs. branch-2. awsClasses = conf.getClasses(AWS_CREDENTIALS_PROVIDER, AWSCredentialsProvider.class); The difference in behavior might be due to HADOOP-7851 , which I notice is in trunk but not branch-2. Steve, would you please take a look? AWSCredentialsProvider can't be used as the default, because it can't be instantiated. testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate) Time elapsed: 1.362 sec <<< ERROR! java.io.IOException: com.amazonaws.auth.AWSCredentialsProvider constructor exception. A class specified in fs.s3a.aws.credentials.provider must provide an accessible constructor accepting URI and Configuration, or an accessible default constructor. at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:319) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388) at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.lang.NoSuchMethodException: com.amazonaws.auth.AWSCredentialsProvider.<init>() at java.lang. Class .getConstructor0( Class .java:3082) at java.lang. Class .getDeclaredConstructor( Class .java:2178) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:315) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388) at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          yes, I see this
          I do suspect it is that patch and will fix.

          I am really surprised that isn't in trunk. What happened there? We should get it into branch-3.0 too

          Show
          stevel@apache.org Steve Loughran added a comment - yes, I see this I do suspect it is that patch and will fix. I am really surprised that isn't in trunk. What happened there? We should get it into branch-3.0 too
          Hide
          cnauroth Chris Nauroth added a comment -

          I am really surprised that isn't in trunk. What happened there? We should get it into branch-3.0 too

          HADOOP-7851 is in trunk and branch-3.0.0-alpha1, but I don't see it in any of the 2.x-based branches. Whether or not that's intentional is hard to say. Its fix version is 0.23.1. In the past, I have found a couple of patches from the 0.23 timeframe that were forgotten in the transition to branch-1 and branch-2.

          Show
          cnauroth Chris Nauroth added a comment - I am really surprised that isn't in trunk. What happened there? We should get it into branch-3.0 too HADOOP-7851 is in trunk and branch-3.0.0-alpha1, but I don't see it in any of the 2.x-based branches. Whether or not that's intentional is hard to say. Its fix version is 0.23.1. In the past, I have found a couple of patches from the 0.23 timeframe that were forgotten in the transition to branch-1 and branch-2.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 006

          fixed to work against trunk. The issue there is that because trunk returns the default value correctly, we can't pass in an interface there. Instead Configuration.getInstances() was copied into S3AUtils and tweaked to only do the list of classes (with assignment check) rather than the full instantiation.

          • adds tests to generate and validate failure modes
          • factors out the error text strings looked for in the tests
          • also fixed test teardown to handle null file context, including in the hadoop common base class.

          Tested: S3 ireland

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 006 fixed to work against trunk. The issue there is that because trunk returns the default value correctly, we can't pass in an interface there. Instead Configuration.getInstances() was copied into S3AUtils and tweaked to only do the list of classes (with assignment check) rather than the full instantiation. adds tests to generate and validate failure modes factors out the error text strings looked for in the tests also fixed test teardown to handle null file context, including in the hadoop common base class. Tested: S3 ireland
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +applied to branch 2 (3-way merge) and tested there. Everything passed

          Show
          stevel@apache.org Steve Loughran added a comment - +applied to branch 2 (3-way merge) and tested there. Everything passed
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 4s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 15s the patch passed
          +1 javac 0m 15s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8)
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 35s the patch passed
          +1 javadoc 0m 12s the patch passed
          +1 unit 0m 15s hadoop-aws in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          14m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824409/HADOOP-13252-006.patch
          JIRA Issue HADOOP-13252
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c468bb702c77 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ae4db25
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 4s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8) +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 35s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 0m 15s hadoop-aws in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 14m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824409/HADOOP-13252-006.patch JIRA Issue HADOOP-13252 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c468bb702c77 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae4db25 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10299/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch 006 tested successfully against both trunk and branch-2. The only problem is the documentation update in core-default.xml got dropped in this revision. I'll be +1 after bringing that back. Let's fix that last round of Checkstyle nitpicks too.

          Show
          cnauroth Chris Nauroth added a comment - Patch 006 tested successfully against both trunk and branch-2. The only problem is the documentation update in core-default.xml got dropped in this revision. I'll be +1 after bringing that back. Let's fix that last round of Checkstyle nitpicks too.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 007; pull in core-default.xml changes and fix checkstyle complaints

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 007; pull in core-default.xml changes and fix checkstyle complaints
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tested on trunk against s3 ireland

          Show
          stevel@apache.org Steve Loughran added a comment - tested on trunk against s3 ireland
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 6m 35s trunk passed
          +1 compile 6m 45s trunk passed
          +1 checkstyle 1m 24s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 6m 47s the patch passed
          +1 javac 6m 47s the patch passed
          -0 checkstyle 1m 29s root: The patch generated 1 new + 92 unchanged - 2 fixed = 93 total (was 94)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 10s the patch passed
          +1 unit 8m 10s hadoop-common in the patch passed.
          +1 unit 0m 22s hadoop-aws in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          67m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824567/HADOOP-13252-007.patch
          JIRA Issue HADOOP-13252
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 1d145c3981fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 091dd19
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 6m 45s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 6m 47s the patch passed +1 javac 6m 47s the patch passed -0 checkstyle 1m 29s root: The patch generated 1 new + 92 unchanged - 2 fixed = 93 total (was 94) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 8m 10s hadoop-common in the patch passed. +1 unit 0m 22s hadoop-aws in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 67m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824567/HADOOP-13252-007.patch JIRA Issue HADOOP-13252 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 1d145c3981fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 091dd19 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10308/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch 007. I did a successful full test run against US-west-2. I have committed this to trunk, branch-2 and branch-2.8. Steve Loughran, thank you for the patch.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch 007. I did a successful full test run against US-west-2. I have committed this to trunk, branch-2 and branch-2.8. Steve Loughran , thank you for the patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10311/)
          HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve (cnauroth: rev 763f0497bb996e331e40caed9ca0af966f5b3fac)

          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AnonymousAWSCredentialsProvider.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java
          • (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3A.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextUtilBase.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
          • (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10311/ ) HADOOP-13252 . Tune S3A provider plugin mechanism. Contributed by Steve (cnauroth: rev 763f0497bb996e331e40caed9ca0af966f5b3fac) (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AnonymousAWSCredentialsProvider.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3A.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextUtilBase.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java
          Hide
          liuml07 Mingliang Liu added a comment -
          294	  /**
          295	   * Create an AWS credential provider.
          296	   * @param conf configuration
          297	   * @param credClass credential class
          298	   * @param uri URI of the FS
          299	   * @return the instantiated class
          300	   * @throws IOException on any instantiation failure.
          301	   */
          302	  static AWSCredentialsProvider createAWSCredentialProvider(
          303	      Configuration conf,
          304	      Class<?> credClass, URI uri) throws IOException {
          305	    AWSCredentialsProvider credentials;
          306	    String className = credClass.getName();
          307	    try {
          308	      LOG.debug("Credential provider class is {}", className);
          309	      try {
          310	        credentials =
          311	            (AWSCredentialsProvider) credClass.getDeclaredConstructor(
          312	                URI.class, Configuration.class).newInstance(uri, conf);
          313	      } catch (NoSuchMethodException | SecurityException e) {
          314	        credentials =
          315	            (AWSCredentialsProvider) credClass.getDeclaredConstructor()
          316	                .newInstance();
          317	      }
          318	    } catch (NoSuchMethodException | SecurityException e) {
          319	      throw new IOException(String.format("%s constructor exception.  A "
          320	          + "class specified in %s must provide an accessible constructor "
          321	          + "accepting URI and Configuration, or an accessible default "
          322	          + "constructor.", className, AWS_CREDENTIALS_PROVIDER), e);
          323	    } catch (ReflectiveOperationException | IllegalArgumentException e) {
          324	      throw new IOException(className + " instantiation exception.", e);
          325	    }
          326	    LOG.debug("Using {} for {}.", credentials, uri);
          327	    return credentials;
          328	  }
          

          It looks like the URI parameter is not needed for our AWSCredentialProvider constructors. Can I file a new JIRA to remove it, if this is true?
          For user:pass in binding URIs, we have

          279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
          280	      credentials.add(new BasicAWSCredentialsProvider(
          281	              creds.getUser(), creds.getPassword()));
          

          This way, we only need configuration object (if necessary) for new AWSCredentialProvider implementations. The benefit is that, if we create AWSCredentialProvider list for DynamoDB, we don't have to pass down the associated file system URI. This might be useful to S3Guard tools.

          Show
          liuml07 Mingliang Liu added a comment - 294 /** 295 * Create an AWS credential provider. 296 * @param conf configuration 297 * @param credClass credential class 298 * @param uri URI of the FS 299 * @ return the instantiated class 300 * @ throws IOException on any instantiation failure. 301 */ 302 static AWSCredentialsProvider createAWSCredentialProvider( 303 Configuration conf, 304 Class <?> credClass, URI uri) throws IOException { 305 AWSCredentialsProvider credentials; 306 String className = credClass.getName(); 307 try { 308 LOG.debug( "Credential provider class is {}" , className); 309 try { 310 credentials = 311 (AWSCredentialsProvider) credClass.getDeclaredConstructor( 312 URI.class, Configuration.class).newInstance(uri, conf); 313 } catch (NoSuchMethodException | SecurityException e) { 314 credentials = 315 (AWSCredentialsProvider) credClass.getDeclaredConstructor() 316 .newInstance(); 317 } 318 } catch (NoSuchMethodException | SecurityException e) { 319 throw new IOException( String .format( "%s constructor exception. A " 320 + "class specified in %s must provide an accessible constructor " 321 + "accepting URI and Configuration, or an accessible default " 322 + "constructor." , className, AWS_CREDENTIALS_PROVIDER), e); 323 } catch (ReflectiveOperationException | IllegalArgumentException e) { 324 throw new IOException(className + " instantiation exception." , e); 325 } 326 LOG.debug( "Using {} for {}." , credentials, uri); 327 return credentials; 328 } It looks like the URI parameter is not needed for our AWSCredentialProvider constructors. Can I file a new JIRA to remove it, if this is true? For user:pass in binding URIs, we have 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf); 280 credentials.add( new BasicAWSCredentialsProvider( 281 creds.getUser(), creds.getPassword())); This way, we only need configuration object (if necessary) for new AWSCredentialProvider implementations. The benefit is that, if we create AWSCredentialProvider list for DynamoDB, we don't have to pass down the associated file system URI. This might be useful to S3Guard tools.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development